Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation 2 ND EDITION, MARCH 2019 by Fay Twersky, Amy Arbreton, Prithi Trivedi
Evaluation Principles and Practices
A Guide to Evaluation at the William and Flora Hewlett Foundation
2ND EDITION MARCH 2019
by Fay Twersky Amy Arbreton Prithi Trivedi
7 PRINCIPLES OF EVALUATION
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
cover photos copy (L to R) Paula BronsteinGetty Images Reportage Tim Peterson OscityShutterstock
The William and Flora Hewlett Foundation is a nonpartisan private charitable foundation that advances ideas and supports institutions to promote a better world Learn more at wwwhewlettorg
Attribution CC BY All text is licensed CC BY photos and images are excluded from this license
TABLE OF CONTENTS
Foreword 1
Introduction 2
The Hewlett Foundationrsquos Seven Principles of Evaluation Practice 5
Roles and Responsibilities 7
Program Staff 7
Grantees 7
The Effective Philanthropy Group 7
Other Departments 8
Evaluation Consultants 9
Practice Guide Planning Implementation and Use 10
PLANNING 10
Beginning Evaluation Planning Early 10
Choosing What to Evaluate Throughout the Strategy Lifecycle 13
Engaging With Grantees 17
Allocating Sufficient Time 18
Clarifying an Evaluationrsquos Purpose 19
Defining Evaluation Questions 22
Identifying Methods 26
Crafting an RFP for an Evaluator 27
Considering Whether or Notmdashand Howmdashto Have the Evaluator Offer Recommendations 28
Choosing an Evaluator and Developing an Agreement 28
IMPLEMENTATION 31
Managing and Staying Involved with an Evaluation 31
Responding to Challenges 33
USE (Including Interpreting and Sharing Findings) 34
Sharing Results Internally 36
Sharing Results Externally 36
Key Terms 38
appendix a Evaluate Throughout a Strategy 41
appendix b Editorial Guidance What Evaluators Need To Know 42
appendix c Engage Grantees In Evaluation 44
appendix d Evaluation Principles 45
End Notes 46
1
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation2nd Edition March 2019
ForewordEvaluation is and has long been an important tool to help us learn and improve our strategies at the William and Flora Hewlett Foundation In 2012 we harmonized our approach to evaluation across our programs with input from staff and external advisors This resulted in our first commonly defined set of Evaluation Principles and Practices1 designed to guide program staff on how to commission purposeful evaluations and use evaluation findings to inform decision making
In 2017 after five years of implementing these principles and practices we took a step back to systemat-ically analyze the foundationrsquos evaluation quality and spending patterns2 Based on our findings which were largely positive but did surface areas for improvement we thought it made sense to update our evaluation guidance The updated guidance in this document reflects lessons from living these princi-ples and insights from the broader field of evaluation As such this document is a revision of the orig-inal Principles and Practices paper and aims to supplement the principlesmdashwhich remain the samemdashwith the practical lessons that have been gained over the past five years
Fay Twersky Amy Arbreton and Prithi Trivedi
Acknowledgements
This paper benefitted from significant input and insights of many people We wish to thank the following Hewlett Foundation staff for their review and comments on earlier drafts of this paper Pat Scheid Angela DeBarger Kerry OrsquoConnor Lori Grange Carla Ganiel Heath Wickline Vidya Krishnamurthy Ruth Levine Larry Kramer Marc Chun Andrea Keller Helsel Eli Sugarman Lindsay Louie Daniel Stid Kent McGuire Jonathan Pershing Emiko Ono and Pooja Raval We would also like to thank the following individuals who served as external re-viewers Megan Colnar Maribel Morey Lisa Ranghelli Patricia Rogers David Devlin-Foltz Susanna Dilliplane Sarah Stachowiak Julia Coffman Scott Chaplowe Meredith Blair Pearlman and Jacob Harold We also thank and acknowledge Karen Lindblom for her contribution to and authorship of the first edition of the principles and practice guidance
2
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
IntroductionEvaluation is part of the fabric of the William and Flora Hewlett Foundation It is referenced in our Guiding Principles3 Evaluation is also an explicit element of our strategy lifecycle (described in Outcome-Focused Philanthropy)4 and is meant to be practiced with frequency intensity and skill across all programs
The purpose of this document is to provide a refreshed description of our philosophy purpose and expectations specifically regarding evaluation at the Hewlett Foundation to clarify staff roles and to outline available support Although the foundationrsquos programs and initiatives differ widely our evalu-ation principles and guidance for practice apply to all of them The principles described in the original version of this paper are enduring but this version of the paper includes additional details as well as updated sections on practicemdashwith new examples and refined guidance
There are many ways that foundation staff learn and measure progressmdashevaluation monitoring and feedback Evaluation is the focus of this paper
What is evaluation
Evaluation is an independent systematic investigation into how why to what extent and for whom out-comes or goals are achieved It can help the foundation answer key questions about strategies substrate-gies clusters of grants and (occasionally) single grants
What is monitoring
Strategy monitoringmdashwhat we call ldquotracking progressrdquo5mdashis a process for checking in on a regular basis to make sure we are on track toward strategy outcomes Tracking progress and evaluation are complementary but different Tracking progress helps keep track of and describe progress toward some longer-term change we want to see whereas evaluation helps us understand why and how change is happening (or not) When tracking progress staff identify the key signs and landmarks that will help them understand if they are on the right track or off course we call these signs ldquoimplementation markersrdquo In contrast evaluation will often draw on progress tracking data but will typically include other methods and data sources to answer more strategic questions
What is feedback
Foundation staff also learn through feedback from the experience of the people who we hope will ulti-mately be positively touched by our work Feedback6 provides new information and insight that can be a valuable source for learning while it complements evaluation and tracking progress it merits focus as a distinct channel
The following paper is organized into three substantive sections (a) Principles (b) Roles and Responsi-bilities and (c) Practice Guide
Our primary audience for this paper is our foundation staff to promote an internal culture of inquiry and practical evaluation In particular program staffmdashas those responsible for planning budgeting commissioning and using third-party evaluationsmdashshould carefully review the guidance and tips presented throughout the practice guide
We share the paper broadlymdashnot as a blueprint but in a spirit of collegialitymdashand with an interest in contributing to othersrsquo efforts and continuing our collective dialogue about evaluation practice
3
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
This document has been updated to reflect many lessons including those from three significant bodies of work the foundation has undertaken since the original principles and practices guidance was developed and adopted in 2012
OUTCOME-FOCUSED PHILANTHROPY
In 2016 the foundation formally adopted Outcome-Focused Philanthro-py (OFP) an evolution of our approach to strategy7 OFP incorporates evaluation into its guidance outlining its importance at every stage of the strategy lifecycle origination implementation refresh and exit OFP reinforces both the importance of evaluation and the strong connections between strategy and evaluation The OFP guidebook states
The usual practice in philanthropy has been to think about evaluation as something to do [only] at the refresh or exit stage In fact evaluation is relevant and important at every stage of the strategy lifecycle Done well it clarifies assumptions contextualizes evidence and helps us learn and adapt as our work proceeds It is useful and important to integrate evaluation planning into the development of a new strategy from the outset Building evaluation into the origination stage provides a proper ldquobaselinerdquo against which to measure subsequent developments prepares staff to collect data in a useful and common format lets grantees know what to expect and when and sets us up to engage in ongoing evalua-tion in the implementation phase
DIVERSITY EQUITY AND INCLUSION WORK
The foundation refined its Guiding Principles in 2016 bringing additional attention and explicit focus to issues of diversity equity and inclusion8 (DEI)mdashboth internally at the foundation and externally in our grant-making and related activities It is now referenced specifically in the Hewlett Foundationrsquos Guiding Principles 9
We seek to promote the values and practice of diversity equity and inclusion in our workforce our culture and our grantmaking When we speak of diversity and inclusion we mean the whole range of attitudes outlooks and perceptions that matter to the people who work with usmdashwhether coming from familiar sources of personal identity like race gender or religion from less common sources that are particular to our institution like place in the foundationrsquos hierarchy or from sources that are idiosyncratic and individual in nature
The foundationrsquos work on diversity equity and inclusion has included a growing number of projects workgroups and activitiesmdashall charged with improving how we consider acknowledge and account for groups that have been marginalized or historically disadvantaged How we practice evaluation at the foundation has been one area of focus through this pro-cess There are three types of considerations we want to make to apply a DEI lens in evaluation first the specific evaluation questions asked includ-
The foundation seeks to promote the values and practice of diversity equity and inclusion in its workforce its culture and its grantmaking
4
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
ing who asks them and whether any address strategy-related issues of diversity equity and inclusion second the process and lens the evaluator uses to gather and interpret different types of data from whom the data are collected and whether and how different stakeholders are engaged in interpreting and using findings and third who conducts the evaluations and their competencies related to understanding the relevant contexts communities issues and tools
AN ASSESSMENT OF THE QUALITY OF OUR EVALUATIONS
In 2017 we took stock of our evaluation practicemdashand found both strong positive developments and areas for improvement10 On the positive side the quality of the evaluations we commission has improved over time They are more rigorous more purpose-driven more useful and used and more widely shared The analysis also uncovered three primary ways in which we can make our evaluations still more valuable
First our evaluations would benefit from more rigor including sharper evaluation questions and more sources of comparisonmdashwhich would strengthen our ability to learn from our evaluations
Second we can and should do much more to engage grantees when planning implementing and using evaluations
Third we need to take greater steps to ensure that we share what we are learning publicly from every evaluation we commissionmdashwhether in full form executive summary or a presentation this is part of our commitment as a foundation to increased openness and transparen-
cy11 as reflected in our foundationrsquos guiding principles12 and allows more
interested parties to learn from the evaluations we have commissioned
The foundation is committed to openness transparency and learning
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
7 PRINCIPLES OF EVALUATION
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
cover photos copy (L to R) Paula BronsteinGetty Images Reportage Tim Peterson OscityShutterstock
The William and Flora Hewlett Foundation is a nonpartisan private charitable foundation that advances ideas and supports institutions to promote a better world Learn more at wwwhewlettorg
Attribution CC BY All text is licensed CC BY photos and images are excluded from this license
TABLE OF CONTENTS
Foreword 1
Introduction 2
The Hewlett Foundationrsquos Seven Principles of Evaluation Practice 5
Roles and Responsibilities 7
Program Staff 7
Grantees 7
The Effective Philanthropy Group 7
Other Departments 8
Evaluation Consultants 9
Practice Guide Planning Implementation and Use 10
PLANNING 10
Beginning Evaluation Planning Early 10
Choosing What to Evaluate Throughout the Strategy Lifecycle 13
Engaging With Grantees 17
Allocating Sufficient Time 18
Clarifying an Evaluationrsquos Purpose 19
Defining Evaluation Questions 22
Identifying Methods 26
Crafting an RFP for an Evaluator 27
Considering Whether or Notmdashand Howmdashto Have the Evaluator Offer Recommendations 28
Choosing an Evaluator and Developing an Agreement 28
IMPLEMENTATION 31
Managing and Staying Involved with an Evaluation 31
Responding to Challenges 33
USE (Including Interpreting and Sharing Findings) 34
Sharing Results Internally 36
Sharing Results Externally 36
Key Terms 38
appendix a Evaluate Throughout a Strategy 41
appendix b Editorial Guidance What Evaluators Need To Know 42
appendix c Engage Grantees In Evaluation 44
appendix d Evaluation Principles 45
End Notes 46
1
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation2nd Edition March 2019
ForewordEvaluation is and has long been an important tool to help us learn and improve our strategies at the William and Flora Hewlett Foundation In 2012 we harmonized our approach to evaluation across our programs with input from staff and external advisors This resulted in our first commonly defined set of Evaluation Principles and Practices1 designed to guide program staff on how to commission purposeful evaluations and use evaluation findings to inform decision making
In 2017 after five years of implementing these principles and practices we took a step back to systemat-ically analyze the foundationrsquos evaluation quality and spending patterns2 Based on our findings which were largely positive but did surface areas for improvement we thought it made sense to update our evaluation guidance The updated guidance in this document reflects lessons from living these princi-ples and insights from the broader field of evaluation As such this document is a revision of the orig-inal Principles and Practices paper and aims to supplement the principlesmdashwhich remain the samemdashwith the practical lessons that have been gained over the past five years
Fay Twersky Amy Arbreton and Prithi Trivedi
Acknowledgements
This paper benefitted from significant input and insights of many people We wish to thank the following Hewlett Foundation staff for their review and comments on earlier drafts of this paper Pat Scheid Angela DeBarger Kerry OrsquoConnor Lori Grange Carla Ganiel Heath Wickline Vidya Krishnamurthy Ruth Levine Larry Kramer Marc Chun Andrea Keller Helsel Eli Sugarman Lindsay Louie Daniel Stid Kent McGuire Jonathan Pershing Emiko Ono and Pooja Raval We would also like to thank the following individuals who served as external re-viewers Megan Colnar Maribel Morey Lisa Ranghelli Patricia Rogers David Devlin-Foltz Susanna Dilliplane Sarah Stachowiak Julia Coffman Scott Chaplowe Meredith Blair Pearlman and Jacob Harold We also thank and acknowledge Karen Lindblom for her contribution to and authorship of the first edition of the principles and practice guidance
2
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
IntroductionEvaluation is part of the fabric of the William and Flora Hewlett Foundation It is referenced in our Guiding Principles3 Evaluation is also an explicit element of our strategy lifecycle (described in Outcome-Focused Philanthropy)4 and is meant to be practiced with frequency intensity and skill across all programs
The purpose of this document is to provide a refreshed description of our philosophy purpose and expectations specifically regarding evaluation at the Hewlett Foundation to clarify staff roles and to outline available support Although the foundationrsquos programs and initiatives differ widely our evalu-ation principles and guidance for practice apply to all of them The principles described in the original version of this paper are enduring but this version of the paper includes additional details as well as updated sections on practicemdashwith new examples and refined guidance
There are many ways that foundation staff learn and measure progressmdashevaluation monitoring and feedback Evaluation is the focus of this paper
What is evaluation
Evaluation is an independent systematic investigation into how why to what extent and for whom out-comes or goals are achieved It can help the foundation answer key questions about strategies substrate-gies clusters of grants and (occasionally) single grants
What is monitoring
Strategy monitoringmdashwhat we call ldquotracking progressrdquo5mdashis a process for checking in on a regular basis to make sure we are on track toward strategy outcomes Tracking progress and evaluation are complementary but different Tracking progress helps keep track of and describe progress toward some longer-term change we want to see whereas evaluation helps us understand why and how change is happening (or not) When tracking progress staff identify the key signs and landmarks that will help them understand if they are on the right track or off course we call these signs ldquoimplementation markersrdquo In contrast evaluation will often draw on progress tracking data but will typically include other methods and data sources to answer more strategic questions
What is feedback
Foundation staff also learn through feedback from the experience of the people who we hope will ulti-mately be positively touched by our work Feedback6 provides new information and insight that can be a valuable source for learning while it complements evaluation and tracking progress it merits focus as a distinct channel
The following paper is organized into three substantive sections (a) Principles (b) Roles and Responsi-bilities and (c) Practice Guide
Our primary audience for this paper is our foundation staff to promote an internal culture of inquiry and practical evaluation In particular program staffmdashas those responsible for planning budgeting commissioning and using third-party evaluationsmdashshould carefully review the guidance and tips presented throughout the practice guide
We share the paper broadlymdashnot as a blueprint but in a spirit of collegialitymdashand with an interest in contributing to othersrsquo efforts and continuing our collective dialogue about evaluation practice
3
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
This document has been updated to reflect many lessons including those from three significant bodies of work the foundation has undertaken since the original principles and practices guidance was developed and adopted in 2012
OUTCOME-FOCUSED PHILANTHROPY
In 2016 the foundation formally adopted Outcome-Focused Philanthro-py (OFP) an evolution of our approach to strategy7 OFP incorporates evaluation into its guidance outlining its importance at every stage of the strategy lifecycle origination implementation refresh and exit OFP reinforces both the importance of evaluation and the strong connections between strategy and evaluation The OFP guidebook states
The usual practice in philanthropy has been to think about evaluation as something to do [only] at the refresh or exit stage In fact evaluation is relevant and important at every stage of the strategy lifecycle Done well it clarifies assumptions contextualizes evidence and helps us learn and adapt as our work proceeds It is useful and important to integrate evaluation planning into the development of a new strategy from the outset Building evaluation into the origination stage provides a proper ldquobaselinerdquo against which to measure subsequent developments prepares staff to collect data in a useful and common format lets grantees know what to expect and when and sets us up to engage in ongoing evalua-tion in the implementation phase
DIVERSITY EQUITY AND INCLUSION WORK
The foundation refined its Guiding Principles in 2016 bringing additional attention and explicit focus to issues of diversity equity and inclusion8 (DEI)mdashboth internally at the foundation and externally in our grant-making and related activities It is now referenced specifically in the Hewlett Foundationrsquos Guiding Principles 9
We seek to promote the values and practice of diversity equity and inclusion in our workforce our culture and our grantmaking When we speak of diversity and inclusion we mean the whole range of attitudes outlooks and perceptions that matter to the people who work with usmdashwhether coming from familiar sources of personal identity like race gender or religion from less common sources that are particular to our institution like place in the foundationrsquos hierarchy or from sources that are idiosyncratic and individual in nature
The foundationrsquos work on diversity equity and inclusion has included a growing number of projects workgroups and activitiesmdashall charged with improving how we consider acknowledge and account for groups that have been marginalized or historically disadvantaged How we practice evaluation at the foundation has been one area of focus through this pro-cess There are three types of considerations we want to make to apply a DEI lens in evaluation first the specific evaluation questions asked includ-
The foundation seeks to promote the values and practice of diversity equity and inclusion in its workforce its culture and its grantmaking
4
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
ing who asks them and whether any address strategy-related issues of diversity equity and inclusion second the process and lens the evaluator uses to gather and interpret different types of data from whom the data are collected and whether and how different stakeholders are engaged in interpreting and using findings and third who conducts the evaluations and their competencies related to understanding the relevant contexts communities issues and tools
AN ASSESSMENT OF THE QUALITY OF OUR EVALUATIONS
In 2017 we took stock of our evaluation practicemdashand found both strong positive developments and areas for improvement10 On the positive side the quality of the evaluations we commission has improved over time They are more rigorous more purpose-driven more useful and used and more widely shared The analysis also uncovered three primary ways in which we can make our evaluations still more valuable
First our evaluations would benefit from more rigor including sharper evaluation questions and more sources of comparisonmdashwhich would strengthen our ability to learn from our evaluations
Second we can and should do much more to engage grantees when planning implementing and using evaluations
Third we need to take greater steps to ensure that we share what we are learning publicly from every evaluation we commissionmdashwhether in full form executive summary or a presentation this is part of our commitment as a foundation to increased openness and transparen-
cy11 as reflected in our foundationrsquos guiding principles12 and allows more
interested parties to learn from the evaluations we have commissioned
The foundation is committed to openness transparency and learning
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
TABLE OF CONTENTS
Foreword 1
Introduction 2
The Hewlett Foundationrsquos Seven Principles of Evaluation Practice 5
Roles and Responsibilities 7
Program Staff 7
Grantees 7
The Effective Philanthropy Group 7
Other Departments 8
Evaluation Consultants 9
Practice Guide Planning Implementation and Use 10
PLANNING 10
Beginning Evaluation Planning Early 10
Choosing What to Evaluate Throughout the Strategy Lifecycle 13
Engaging With Grantees 17
Allocating Sufficient Time 18
Clarifying an Evaluationrsquos Purpose 19
Defining Evaluation Questions 22
Identifying Methods 26
Crafting an RFP for an Evaluator 27
Considering Whether or Notmdashand Howmdashto Have the Evaluator Offer Recommendations 28
Choosing an Evaluator and Developing an Agreement 28
IMPLEMENTATION 31
Managing and Staying Involved with an Evaluation 31
Responding to Challenges 33
USE (Including Interpreting and Sharing Findings) 34
Sharing Results Internally 36
Sharing Results Externally 36
Key Terms 38
appendix a Evaluate Throughout a Strategy 41
appendix b Editorial Guidance What Evaluators Need To Know 42
appendix c Engage Grantees In Evaluation 44
appendix d Evaluation Principles 45
End Notes 46
1
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation2nd Edition March 2019
ForewordEvaluation is and has long been an important tool to help us learn and improve our strategies at the William and Flora Hewlett Foundation In 2012 we harmonized our approach to evaluation across our programs with input from staff and external advisors This resulted in our first commonly defined set of Evaluation Principles and Practices1 designed to guide program staff on how to commission purposeful evaluations and use evaluation findings to inform decision making
In 2017 after five years of implementing these principles and practices we took a step back to systemat-ically analyze the foundationrsquos evaluation quality and spending patterns2 Based on our findings which were largely positive but did surface areas for improvement we thought it made sense to update our evaluation guidance The updated guidance in this document reflects lessons from living these princi-ples and insights from the broader field of evaluation As such this document is a revision of the orig-inal Principles and Practices paper and aims to supplement the principlesmdashwhich remain the samemdashwith the practical lessons that have been gained over the past five years
Fay Twersky Amy Arbreton and Prithi Trivedi
Acknowledgements
This paper benefitted from significant input and insights of many people We wish to thank the following Hewlett Foundation staff for their review and comments on earlier drafts of this paper Pat Scheid Angela DeBarger Kerry OrsquoConnor Lori Grange Carla Ganiel Heath Wickline Vidya Krishnamurthy Ruth Levine Larry Kramer Marc Chun Andrea Keller Helsel Eli Sugarman Lindsay Louie Daniel Stid Kent McGuire Jonathan Pershing Emiko Ono and Pooja Raval We would also like to thank the following individuals who served as external re-viewers Megan Colnar Maribel Morey Lisa Ranghelli Patricia Rogers David Devlin-Foltz Susanna Dilliplane Sarah Stachowiak Julia Coffman Scott Chaplowe Meredith Blair Pearlman and Jacob Harold We also thank and acknowledge Karen Lindblom for her contribution to and authorship of the first edition of the principles and practice guidance
2
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
IntroductionEvaluation is part of the fabric of the William and Flora Hewlett Foundation It is referenced in our Guiding Principles3 Evaluation is also an explicit element of our strategy lifecycle (described in Outcome-Focused Philanthropy)4 and is meant to be practiced with frequency intensity and skill across all programs
The purpose of this document is to provide a refreshed description of our philosophy purpose and expectations specifically regarding evaluation at the Hewlett Foundation to clarify staff roles and to outline available support Although the foundationrsquos programs and initiatives differ widely our evalu-ation principles and guidance for practice apply to all of them The principles described in the original version of this paper are enduring but this version of the paper includes additional details as well as updated sections on practicemdashwith new examples and refined guidance
There are many ways that foundation staff learn and measure progressmdashevaluation monitoring and feedback Evaluation is the focus of this paper
What is evaluation
Evaluation is an independent systematic investigation into how why to what extent and for whom out-comes or goals are achieved It can help the foundation answer key questions about strategies substrate-gies clusters of grants and (occasionally) single grants
What is monitoring
Strategy monitoringmdashwhat we call ldquotracking progressrdquo5mdashis a process for checking in on a regular basis to make sure we are on track toward strategy outcomes Tracking progress and evaluation are complementary but different Tracking progress helps keep track of and describe progress toward some longer-term change we want to see whereas evaluation helps us understand why and how change is happening (or not) When tracking progress staff identify the key signs and landmarks that will help them understand if they are on the right track or off course we call these signs ldquoimplementation markersrdquo In contrast evaluation will often draw on progress tracking data but will typically include other methods and data sources to answer more strategic questions
What is feedback
Foundation staff also learn through feedback from the experience of the people who we hope will ulti-mately be positively touched by our work Feedback6 provides new information and insight that can be a valuable source for learning while it complements evaluation and tracking progress it merits focus as a distinct channel
The following paper is organized into three substantive sections (a) Principles (b) Roles and Responsi-bilities and (c) Practice Guide
Our primary audience for this paper is our foundation staff to promote an internal culture of inquiry and practical evaluation In particular program staffmdashas those responsible for planning budgeting commissioning and using third-party evaluationsmdashshould carefully review the guidance and tips presented throughout the practice guide
We share the paper broadlymdashnot as a blueprint but in a spirit of collegialitymdashand with an interest in contributing to othersrsquo efforts and continuing our collective dialogue about evaluation practice
3
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
This document has been updated to reflect many lessons including those from three significant bodies of work the foundation has undertaken since the original principles and practices guidance was developed and adopted in 2012
OUTCOME-FOCUSED PHILANTHROPY
In 2016 the foundation formally adopted Outcome-Focused Philanthro-py (OFP) an evolution of our approach to strategy7 OFP incorporates evaluation into its guidance outlining its importance at every stage of the strategy lifecycle origination implementation refresh and exit OFP reinforces both the importance of evaluation and the strong connections between strategy and evaluation The OFP guidebook states
The usual practice in philanthropy has been to think about evaluation as something to do [only] at the refresh or exit stage In fact evaluation is relevant and important at every stage of the strategy lifecycle Done well it clarifies assumptions contextualizes evidence and helps us learn and adapt as our work proceeds It is useful and important to integrate evaluation planning into the development of a new strategy from the outset Building evaluation into the origination stage provides a proper ldquobaselinerdquo against which to measure subsequent developments prepares staff to collect data in a useful and common format lets grantees know what to expect and when and sets us up to engage in ongoing evalua-tion in the implementation phase
DIVERSITY EQUITY AND INCLUSION WORK
The foundation refined its Guiding Principles in 2016 bringing additional attention and explicit focus to issues of diversity equity and inclusion8 (DEI)mdashboth internally at the foundation and externally in our grant-making and related activities It is now referenced specifically in the Hewlett Foundationrsquos Guiding Principles 9
We seek to promote the values and practice of diversity equity and inclusion in our workforce our culture and our grantmaking When we speak of diversity and inclusion we mean the whole range of attitudes outlooks and perceptions that matter to the people who work with usmdashwhether coming from familiar sources of personal identity like race gender or religion from less common sources that are particular to our institution like place in the foundationrsquos hierarchy or from sources that are idiosyncratic and individual in nature
The foundationrsquos work on diversity equity and inclusion has included a growing number of projects workgroups and activitiesmdashall charged with improving how we consider acknowledge and account for groups that have been marginalized or historically disadvantaged How we practice evaluation at the foundation has been one area of focus through this pro-cess There are three types of considerations we want to make to apply a DEI lens in evaluation first the specific evaluation questions asked includ-
The foundation seeks to promote the values and practice of diversity equity and inclusion in its workforce its culture and its grantmaking
4
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
ing who asks them and whether any address strategy-related issues of diversity equity and inclusion second the process and lens the evaluator uses to gather and interpret different types of data from whom the data are collected and whether and how different stakeholders are engaged in interpreting and using findings and third who conducts the evaluations and their competencies related to understanding the relevant contexts communities issues and tools
AN ASSESSMENT OF THE QUALITY OF OUR EVALUATIONS
In 2017 we took stock of our evaluation practicemdashand found both strong positive developments and areas for improvement10 On the positive side the quality of the evaluations we commission has improved over time They are more rigorous more purpose-driven more useful and used and more widely shared The analysis also uncovered three primary ways in which we can make our evaluations still more valuable
First our evaluations would benefit from more rigor including sharper evaluation questions and more sources of comparisonmdashwhich would strengthen our ability to learn from our evaluations
Second we can and should do much more to engage grantees when planning implementing and using evaluations
Third we need to take greater steps to ensure that we share what we are learning publicly from every evaluation we commissionmdashwhether in full form executive summary or a presentation this is part of our commitment as a foundation to increased openness and transparen-
cy11 as reflected in our foundationrsquos guiding principles12 and allows more
interested parties to learn from the evaluations we have commissioned
The foundation is committed to openness transparency and learning
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
1
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation2nd Edition March 2019
ForewordEvaluation is and has long been an important tool to help us learn and improve our strategies at the William and Flora Hewlett Foundation In 2012 we harmonized our approach to evaluation across our programs with input from staff and external advisors This resulted in our first commonly defined set of Evaluation Principles and Practices1 designed to guide program staff on how to commission purposeful evaluations and use evaluation findings to inform decision making
In 2017 after five years of implementing these principles and practices we took a step back to systemat-ically analyze the foundationrsquos evaluation quality and spending patterns2 Based on our findings which were largely positive but did surface areas for improvement we thought it made sense to update our evaluation guidance The updated guidance in this document reflects lessons from living these princi-ples and insights from the broader field of evaluation As such this document is a revision of the orig-inal Principles and Practices paper and aims to supplement the principlesmdashwhich remain the samemdashwith the practical lessons that have been gained over the past five years
Fay Twersky Amy Arbreton and Prithi Trivedi
Acknowledgements
This paper benefitted from significant input and insights of many people We wish to thank the following Hewlett Foundation staff for their review and comments on earlier drafts of this paper Pat Scheid Angela DeBarger Kerry OrsquoConnor Lori Grange Carla Ganiel Heath Wickline Vidya Krishnamurthy Ruth Levine Larry Kramer Marc Chun Andrea Keller Helsel Eli Sugarman Lindsay Louie Daniel Stid Kent McGuire Jonathan Pershing Emiko Ono and Pooja Raval We would also like to thank the following individuals who served as external re-viewers Megan Colnar Maribel Morey Lisa Ranghelli Patricia Rogers David Devlin-Foltz Susanna Dilliplane Sarah Stachowiak Julia Coffman Scott Chaplowe Meredith Blair Pearlman and Jacob Harold We also thank and acknowledge Karen Lindblom for her contribution to and authorship of the first edition of the principles and practice guidance
2
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
IntroductionEvaluation is part of the fabric of the William and Flora Hewlett Foundation It is referenced in our Guiding Principles3 Evaluation is also an explicit element of our strategy lifecycle (described in Outcome-Focused Philanthropy)4 and is meant to be practiced with frequency intensity and skill across all programs
The purpose of this document is to provide a refreshed description of our philosophy purpose and expectations specifically regarding evaluation at the Hewlett Foundation to clarify staff roles and to outline available support Although the foundationrsquos programs and initiatives differ widely our evalu-ation principles and guidance for practice apply to all of them The principles described in the original version of this paper are enduring but this version of the paper includes additional details as well as updated sections on practicemdashwith new examples and refined guidance
There are many ways that foundation staff learn and measure progressmdashevaluation monitoring and feedback Evaluation is the focus of this paper
What is evaluation
Evaluation is an independent systematic investigation into how why to what extent and for whom out-comes or goals are achieved It can help the foundation answer key questions about strategies substrate-gies clusters of grants and (occasionally) single grants
What is monitoring
Strategy monitoringmdashwhat we call ldquotracking progressrdquo5mdashis a process for checking in on a regular basis to make sure we are on track toward strategy outcomes Tracking progress and evaluation are complementary but different Tracking progress helps keep track of and describe progress toward some longer-term change we want to see whereas evaluation helps us understand why and how change is happening (or not) When tracking progress staff identify the key signs and landmarks that will help them understand if they are on the right track or off course we call these signs ldquoimplementation markersrdquo In contrast evaluation will often draw on progress tracking data but will typically include other methods and data sources to answer more strategic questions
What is feedback
Foundation staff also learn through feedback from the experience of the people who we hope will ulti-mately be positively touched by our work Feedback6 provides new information and insight that can be a valuable source for learning while it complements evaluation and tracking progress it merits focus as a distinct channel
The following paper is organized into three substantive sections (a) Principles (b) Roles and Responsi-bilities and (c) Practice Guide
Our primary audience for this paper is our foundation staff to promote an internal culture of inquiry and practical evaluation In particular program staffmdashas those responsible for planning budgeting commissioning and using third-party evaluationsmdashshould carefully review the guidance and tips presented throughout the practice guide
We share the paper broadlymdashnot as a blueprint but in a spirit of collegialitymdashand with an interest in contributing to othersrsquo efforts and continuing our collective dialogue about evaluation practice
3
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
This document has been updated to reflect many lessons including those from three significant bodies of work the foundation has undertaken since the original principles and practices guidance was developed and adopted in 2012
OUTCOME-FOCUSED PHILANTHROPY
In 2016 the foundation formally adopted Outcome-Focused Philanthro-py (OFP) an evolution of our approach to strategy7 OFP incorporates evaluation into its guidance outlining its importance at every stage of the strategy lifecycle origination implementation refresh and exit OFP reinforces both the importance of evaluation and the strong connections between strategy and evaluation The OFP guidebook states
The usual practice in philanthropy has been to think about evaluation as something to do [only] at the refresh or exit stage In fact evaluation is relevant and important at every stage of the strategy lifecycle Done well it clarifies assumptions contextualizes evidence and helps us learn and adapt as our work proceeds It is useful and important to integrate evaluation planning into the development of a new strategy from the outset Building evaluation into the origination stage provides a proper ldquobaselinerdquo against which to measure subsequent developments prepares staff to collect data in a useful and common format lets grantees know what to expect and when and sets us up to engage in ongoing evalua-tion in the implementation phase
DIVERSITY EQUITY AND INCLUSION WORK
The foundation refined its Guiding Principles in 2016 bringing additional attention and explicit focus to issues of diversity equity and inclusion8 (DEI)mdashboth internally at the foundation and externally in our grant-making and related activities It is now referenced specifically in the Hewlett Foundationrsquos Guiding Principles 9
We seek to promote the values and practice of diversity equity and inclusion in our workforce our culture and our grantmaking When we speak of diversity and inclusion we mean the whole range of attitudes outlooks and perceptions that matter to the people who work with usmdashwhether coming from familiar sources of personal identity like race gender or religion from less common sources that are particular to our institution like place in the foundationrsquos hierarchy or from sources that are idiosyncratic and individual in nature
The foundationrsquos work on diversity equity and inclusion has included a growing number of projects workgroups and activitiesmdashall charged with improving how we consider acknowledge and account for groups that have been marginalized or historically disadvantaged How we practice evaluation at the foundation has been one area of focus through this pro-cess There are three types of considerations we want to make to apply a DEI lens in evaluation first the specific evaluation questions asked includ-
The foundation seeks to promote the values and practice of diversity equity and inclusion in its workforce its culture and its grantmaking
4
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
ing who asks them and whether any address strategy-related issues of diversity equity and inclusion second the process and lens the evaluator uses to gather and interpret different types of data from whom the data are collected and whether and how different stakeholders are engaged in interpreting and using findings and third who conducts the evaluations and their competencies related to understanding the relevant contexts communities issues and tools
AN ASSESSMENT OF THE QUALITY OF OUR EVALUATIONS
In 2017 we took stock of our evaluation practicemdashand found both strong positive developments and areas for improvement10 On the positive side the quality of the evaluations we commission has improved over time They are more rigorous more purpose-driven more useful and used and more widely shared The analysis also uncovered three primary ways in which we can make our evaluations still more valuable
First our evaluations would benefit from more rigor including sharper evaluation questions and more sources of comparisonmdashwhich would strengthen our ability to learn from our evaluations
Second we can and should do much more to engage grantees when planning implementing and using evaluations
Third we need to take greater steps to ensure that we share what we are learning publicly from every evaluation we commissionmdashwhether in full form executive summary or a presentation this is part of our commitment as a foundation to increased openness and transparen-
cy11 as reflected in our foundationrsquos guiding principles12 and allows more
interested parties to learn from the evaluations we have commissioned
The foundation is committed to openness transparency and learning
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
2
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
IntroductionEvaluation is part of the fabric of the William and Flora Hewlett Foundation It is referenced in our Guiding Principles3 Evaluation is also an explicit element of our strategy lifecycle (described in Outcome-Focused Philanthropy)4 and is meant to be practiced with frequency intensity and skill across all programs
The purpose of this document is to provide a refreshed description of our philosophy purpose and expectations specifically regarding evaluation at the Hewlett Foundation to clarify staff roles and to outline available support Although the foundationrsquos programs and initiatives differ widely our evalu-ation principles and guidance for practice apply to all of them The principles described in the original version of this paper are enduring but this version of the paper includes additional details as well as updated sections on practicemdashwith new examples and refined guidance
There are many ways that foundation staff learn and measure progressmdashevaluation monitoring and feedback Evaluation is the focus of this paper
What is evaluation
Evaluation is an independent systematic investigation into how why to what extent and for whom out-comes or goals are achieved It can help the foundation answer key questions about strategies substrate-gies clusters of grants and (occasionally) single grants
What is monitoring
Strategy monitoringmdashwhat we call ldquotracking progressrdquo5mdashis a process for checking in on a regular basis to make sure we are on track toward strategy outcomes Tracking progress and evaluation are complementary but different Tracking progress helps keep track of and describe progress toward some longer-term change we want to see whereas evaluation helps us understand why and how change is happening (or not) When tracking progress staff identify the key signs and landmarks that will help them understand if they are on the right track or off course we call these signs ldquoimplementation markersrdquo In contrast evaluation will often draw on progress tracking data but will typically include other methods and data sources to answer more strategic questions
What is feedback
Foundation staff also learn through feedback from the experience of the people who we hope will ulti-mately be positively touched by our work Feedback6 provides new information and insight that can be a valuable source for learning while it complements evaluation and tracking progress it merits focus as a distinct channel
The following paper is organized into three substantive sections (a) Principles (b) Roles and Responsi-bilities and (c) Practice Guide
Our primary audience for this paper is our foundation staff to promote an internal culture of inquiry and practical evaluation In particular program staffmdashas those responsible for planning budgeting commissioning and using third-party evaluationsmdashshould carefully review the guidance and tips presented throughout the practice guide
We share the paper broadlymdashnot as a blueprint but in a spirit of collegialitymdashand with an interest in contributing to othersrsquo efforts and continuing our collective dialogue about evaluation practice
3
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
This document has been updated to reflect many lessons including those from three significant bodies of work the foundation has undertaken since the original principles and practices guidance was developed and adopted in 2012
OUTCOME-FOCUSED PHILANTHROPY
In 2016 the foundation formally adopted Outcome-Focused Philanthro-py (OFP) an evolution of our approach to strategy7 OFP incorporates evaluation into its guidance outlining its importance at every stage of the strategy lifecycle origination implementation refresh and exit OFP reinforces both the importance of evaluation and the strong connections between strategy and evaluation The OFP guidebook states
The usual practice in philanthropy has been to think about evaluation as something to do [only] at the refresh or exit stage In fact evaluation is relevant and important at every stage of the strategy lifecycle Done well it clarifies assumptions contextualizes evidence and helps us learn and adapt as our work proceeds It is useful and important to integrate evaluation planning into the development of a new strategy from the outset Building evaluation into the origination stage provides a proper ldquobaselinerdquo against which to measure subsequent developments prepares staff to collect data in a useful and common format lets grantees know what to expect and when and sets us up to engage in ongoing evalua-tion in the implementation phase
DIVERSITY EQUITY AND INCLUSION WORK
The foundation refined its Guiding Principles in 2016 bringing additional attention and explicit focus to issues of diversity equity and inclusion8 (DEI)mdashboth internally at the foundation and externally in our grant-making and related activities It is now referenced specifically in the Hewlett Foundationrsquos Guiding Principles 9
We seek to promote the values and practice of diversity equity and inclusion in our workforce our culture and our grantmaking When we speak of diversity and inclusion we mean the whole range of attitudes outlooks and perceptions that matter to the people who work with usmdashwhether coming from familiar sources of personal identity like race gender or religion from less common sources that are particular to our institution like place in the foundationrsquos hierarchy or from sources that are idiosyncratic and individual in nature
The foundationrsquos work on diversity equity and inclusion has included a growing number of projects workgroups and activitiesmdashall charged with improving how we consider acknowledge and account for groups that have been marginalized or historically disadvantaged How we practice evaluation at the foundation has been one area of focus through this pro-cess There are three types of considerations we want to make to apply a DEI lens in evaluation first the specific evaluation questions asked includ-
The foundation seeks to promote the values and practice of diversity equity and inclusion in its workforce its culture and its grantmaking
4
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
ing who asks them and whether any address strategy-related issues of diversity equity and inclusion second the process and lens the evaluator uses to gather and interpret different types of data from whom the data are collected and whether and how different stakeholders are engaged in interpreting and using findings and third who conducts the evaluations and their competencies related to understanding the relevant contexts communities issues and tools
AN ASSESSMENT OF THE QUALITY OF OUR EVALUATIONS
In 2017 we took stock of our evaluation practicemdashand found both strong positive developments and areas for improvement10 On the positive side the quality of the evaluations we commission has improved over time They are more rigorous more purpose-driven more useful and used and more widely shared The analysis also uncovered three primary ways in which we can make our evaluations still more valuable
First our evaluations would benefit from more rigor including sharper evaluation questions and more sources of comparisonmdashwhich would strengthen our ability to learn from our evaluations
Second we can and should do much more to engage grantees when planning implementing and using evaluations
Third we need to take greater steps to ensure that we share what we are learning publicly from every evaluation we commissionmdashwhether in full form executive summary or a presentation this is part of our commitment as a foundation to increased openness and transparen-
cy11 as reflected in our foundationrsquos guiding principles12 and allows more
interested parties to learn from the evaluations we have commissioned
The foundation is committed to openness transparency and learning
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
3
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
This document has been updated to reflect many lessons including those from three significant bodies of work the foundation has undertaken since the original principles and practices guidance was developed and adopted in 2012
OUTCOME-FOCUSED PHILANTHROPY
In 2016 the foundation formally adopted Outcome-Focused Philanthro-py (OFP) an evolution of our approach to strategy7 OFP incorporates evaluation into its guidance outlining its importance at every stage of the strategy lifecycle origination implementation refresh and exit OFP reinforces both the importance of evaluation and the strong connections between strategy and evaluation The OFP guidebook states
The usual practice in philanthropy has been to think about evaluation as something to do [only] at the refresh or exit stage In fact evaluation is relevant and important at every stage of the strategy lifecycle Done well it clarifies assumptions contextualizes evidence and helps us learn and adapt as our work proceeds It is useful and important to integrate evaluation planning into the development of a new strategy from the outset Building evaluation into the origination stage provides a proper ldquobaselinerdquo against which to measure subsequent developments prepares staff to collect data in a useful and common format lets grantees know what to expect and when and sets us up to engage in ongoing evalua-tion in the implementation phase
DIVERSITY EQUITY AND INCLUSION WORK
The foundation refined its Guiding Principles in 2016 bringing additional attention and explicit focus to issues of diversity equity and inclusion8 (DEI)mdashboth internally at the foundation and externally in our grant-making and related activities It is now referenced specifically in the Hewlett Foundationrsquos Guiding Principles 9
We seek to promote the values and practice of diversity equity and inclusion in our workforce our culture and our grantmaking When we speak of diversity and inclusion we mean the whole range of attitudes outlooks and perceptions that matter to the people who work with usmdashwhether coming from familiar sources of personal identity like race gender or religion from less common sources that are particular to our institution like place in the foundationrsquos hierarchy or from sources that are idiosyncratic and individual in nature
The foundationrsquos work on diversity equity and inclusion has included a growing number of projects workgroups and activitiesmdashall charged with improving how we consider acknowledge and account for groups that have been marginalized or historically disadvantaged How we practice evaluation at the foundation has been one area of focus through this pro-cess There are three types of considerations we want to make to apply a DEI lens in evaluation first the specific evaluation questions asked includ-
The foundation seeks to promote the values and practice of diversity equity and inclusion in its workforce its culture and its grantmaking
4
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
ing who asks them and whether any address strategy-related issues of diversity equity and inclusion second the process and lens the evaluator uses to gather and interpret different types of data from whom the data are collected and whether and how different stakeholders are engaged in interpreting and using findings and third who conducts the evaluations and their competencies related to understanding the relevant contexts communities issues and tools
AN ASSESSMENT OF THE QUALITY OF OUR EVALUATIONS
In 2017 we took stock of our evaluation practicemdashand found both strong positive developments and areas for improvement10 On the positive side the quality of the evaluations we commission has improved over time They are more rigorous more purpose-driven more useful and used and more widely shared The analysis also uncovered three primary ways in which we can make our evaluations still more valuable
First our evaluations would benefit from more rigor including sharper evaluation questions and more sources of comparisonmdashwhich would strengthen our ability to learn from our evaluations
Second we can and should do much more to engage grantees when planning implementing and using evaluations
Third we need to take greater steps to ensure that we share what we are learning publicly from every evaluation we commissionmdashwhether in full form executive summary or a presentation this is part of our commitment as a foundation to increased openness and transparen-
cy11 as reflected in our foundationrsquos guiding principles12 and allows more
interested parties to learn from the evaluations we have commissioned
The foundation is committed to openness transparency and learning
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
4
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
ing who asks them and whether any address strategy-related issues of diversity equity and inclusion second the process and lens the evaluator uses to gather and interpret different types of data from whom the data are collected and whether and how different stakeholders are engaged in interpreting and using findings and third who conducts the evaluations and their competencies related to understanding the relevant contexts communities issues and tools
AN ASSESSMENT OF THE QUALITY OF OUR EVALUATIONS
In 2017 we took stock of our evaluation practicemdashand found both strong positive developments and areas for improvement10 On the positive side the quality of the evaluations we commission has improved over time They are more rigorous more purpose-driven more useful and used and more widely shared The analysis also uncovered three primary ways in which we can make our evaluations still more valuable
First our evaluations would benefit from more rigor including sharper evaluation questions and more sources of comparisonmdashwhich would strengthen our ability to learn from our evaluations
Second we can and should do much more to engage grantees when planning implementing and using evaluations
Third we need to take greater steps to ensure that we share what we are learning publicly from every evaluation we commissionmdashwhether in full form executive summary or a presentation this is part of our commitment as a foundation to increased openness and transparen-
cy11 as reflected in our foundationrsquos guiding principles12 and allows more
interested parties to learn from the evaluations we have commissioned
The foundation is committed to openness transparency and learning
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
5
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
The Hewlett Foundationrsquos Seven Principles of Evaluation PracticeWe aspire to have the following principles guide our evaluation practice
1 WE LEAD WITH PURPOSE We design evaluation with actions and decisions in mind We ask ldquoHow and when will we (and oth-ers) use the information that comes from this evaluationrdquo By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used It is all too common for evaluations to be commissioned without a clear purpose and then to be shelved without generating useful insights
2 EVALUATION IS FUNDAMENTALLY A LEARNING PROCESS As we engage in evaluation planning implementation and use of results we actively learn and adapt Evaluative thinking and planning inform strategy and target-setting They help clarify evidence and assumptions that undergird the approach to our work Establishing evaluation questions helps us make visible and refine our thinking about how why to what extent for whom and when outcomes or goals are expected to be achieved As we implement our strategies we use evaluation as a key ve-hicle for learning bringing new insights to our work and to the work of others A key part of learning from evaluation is taking time to reflect on findings and to ask ldquoNow whatrdquo13
1 We lead with purpose
7 We use the data
2 Evaluation is fundamentally a learning process
4 We strategically choose what to evaluate
3 Evaluation is an explicit and key part of the strategy lifecycle
5 We choose methods that maximize rigor without compromising relevance
6 We share our findings with appropriate audiences and publicly
7 PRINCIPLES OF EVALUATION
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
6
Evaluation Principles and Practices The Hewlett Foundationrsquos Seven Principles of Evaluation Practice
3 WE TREAT EVALUATION AS AN EXPLICIT AND KEY PART OF THE STRATEGY LIFECYCLE Building evaluative thinking into strategy does two things (a) it helps articulate key assumptions and logical (or illogical) connections in a theory of change and (b) it encourages us to establish a starting point for evaluation questions and propose a way to answer those questions in a practical meaning-ful sequence with actions and decisions in mind throughout the strategy lifecycle In practice this can take many forms such as a planned series of periodic evaluations of substrategies or clusters of grantees a developmental evaluation that is ongoing a formative or summative evaluation planned for a key juncture or a summary of lessons learned at a planned exit
4 WE CANNOT EVALUATE EVERYTHING SO WE CHOOSE STRATEGICALLY Several criteria guide decisions about where to put our evaluation-related time and dollars including urgency to consider course corrections or future funding decisions the opportunity for learning the potential for strategic or reputational risk and size of investment as a proxy for importance Within these parameters every strategy or a key part of every strategy will have an evaluation underway within three years of origination or refresh Planning for an evaluation within such a timeframe en-sures that we do not go too long without getting an external third-party perspective on how well (or not) the work is going and whether any unexpected issues have arisen
5 WE CHOOSE METHODS OF MEASUREMENT THAT ALLOW US TO MAXIMIZE RIGOR WITHOUT COMPROMISING RELEVANCE We match methods to questions and do not choose one approach or privilege one method over oth-ers We select evaluation designs that use multiple methods and data sources when possible in order to strengthen our evaluation design and reduce bias All evaluations clearly articulate methods used and the limitations of those methods Evaluations include comparative reference points or methods to help us assess how well we are doing compared with our own and othersrsquo expectationsmdashover time and across types of interventions organizations populations or regions
6 WE SHARE WHAT WE ARE LEARNING WITH APPROPRIATE AUDIENCES AND SHARE PUBLICLY SO THAT OTHERS CAN LEARN AS WELL As we plan evaluations we consider and identify audiences for the findings We communicate early with our grantees and co-funders about our intention to evaluate and our plans to share findings and we involve them as appropriate in issues of planning design and interpretation We presumptively share the results of our evaluations so that others may learn from our successes and failures We will make principled exceptions on a case-by-case basis about whether to share a full report executive summary or presentation from an evaluation with care given to issues of confidentiality and not wanting to cause harm
7 WE USE THE DATA Not using our findings is a missed opportunity for learning improvement and course correctionmdashand a waste of time energy and resources It is imperative that we take time to reflect on the evalu-ation results generate implications for our strategies grantees policy andor practice and adapt as appropriate We recognize the value in combining the insights from evaluation results with our own experiences We support our grantees in doing the same
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
7
Evaluation Principles and Practices Roles and Responsibilities
Roles and ResponsibilitiesMany people are involved in evaluations Programs have the primary responsibility for evaluations They are responsible for building evaluations into their strategy identifying what and when they will evaluate and commissioning managing and using findings from the evaluations The evalu-ation officer in the Effective Philanthropy Group is available to support program staff Grants manage-ment legal communications and finance staff also have roles in the process
PROGRAM STAFF
Program officers or directors play the leading role in planning for when and what aspects of their strate-gies to evaluate for commissioning evaluations and for managing them from planning and implement-ing to using and sharing Commissioners are the primary point people for coordinating evalua-tions and they must carefully review evaluation reports executive summaries and presentations before public release They are responsible for checking for sensitive information that may not be appropriate for public sharing and making sure that the evaluator has accurately described for example advocacy work with an appropriate disclaimer (see Appendix B for examples) The eval-uation commissioner is responsible for managing adherence to the foundationrsquos ldquoEvaluation Checklist Components for a Successful Evaluationrdquo
Most commonly program associates create a system for accessing data internal to the foundation (such as background documents and grant reports) to share with the evaluator This is an important role and can be time-consuming at key junctures of the evaluation process The program associates also typical-ly assist with review and selection of evaluators support contract development and monitoring of the contract and help organize and participate in workshops (eg with grantees to discuss findings and with evaluation advisory groups)
Some programs teams have selected one team member to manage their evaluations This person coordi-nates with other members of a strategy team and acts as the point person with evaluation consultants (For example the program director for the Madison Initiative manages evaluations for that team which consists of the director two program officers and a program associate A Global Development and Population Program Officer manages the formative evaluation of the Transparency Participation and Accountability strategy working with a team of four program officers and three program associates) Other programs have individual staff manage their own substrategy or grant cluster evaluations
GRANTEES
Grantees are involved as participants and often as intended audience and users of evaluation findings Grantees should be informed about plans for evaluation as early as possible and should be includedmdashto the extent reasonable and feasiblemdashthroughout the planning implementation and use (and sharing) phases of an evaluation (See Appendix C for guidance on engaging grantees)
THE EFFECTIVE PHILANTHROPY GROUP
As the foundationrsquos approach to evaluation has become more deliberate and systematic the founda-tionrsquos leadership has come to appreciate the value and timeliness of expert support for evaluation Therefore as part of its Effective Philanthropy Group (EPG) in 2013 the foundation created a central support function for programsrsquo evaluation efforts
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
8
Evaluation Principles and Practices Roles and Responsibilities
EPG staff act as the Hewlett Foundationrsquos in-house experts on philanthropic practice combining insti-tutional knowledge and technical assistance to help program staff be more effective grantmakers In the case of the evaluation officerrsquos role this includes all things related to evaluation as well as coordinating effectively as needed with the other internal EPG officersmdashStrategy Organizational Learning and Orga-nizational Effectivenessmdashon any overlapping areas of learning assessment and training
Programs are not technically required to use the evaluation officerrsquos support however it is a central form of support the foundation makes available to any program staff seeking evaluation assistance The evaluation officerrsquos role currently includes the following
Internal Consulting to Program Staff The bulk of the evaluation officerrsquos time is spent on internal consulting to support program staff as they work on their evaluations Like the other EPG staff the evaluation officer provides support in three main ways
bull Resource Provider Maintain updated practical resources to share with programs as examples and templates for each step in the evaluation process eg an ldquoEvaluation Checklistrdquo one-pager and examples of evaluation planning tools requests for proposals (RFPs) evaluation designs and evaluation products
bull Ad-Hoc Advisor On a periodic and as-requested basis step in and support program staff eg in framing evaluation priorities questions sequencing and methods in development of RFPs and review of proposals and in supporting internal and external sharing of resultsmdashcoordinating with relevant program legal and communications staff as well as grantees and other external partners
bull Guide-by-the-Side When needed provide deep ongoing support to program staff throughout an evaluation
If program staff are unsure what type of support is needed they can ask the evaluation officer to re-view options
Institutional Knowledge of Evaluation Principles and Practices The evaluation officer is responsible for keeping the principles and practices up to date serving as the internal source for all questions (internal and external) related to evaluation practice at the foundation tracking evaluation quality and spending (with assistance from the finance department) and orienting and training staff in the foundationrsquos principles and practices guidance As a centralized function in a foundation with decentralized program staff the evaluation officer also plays a role in sharing relevant lessons across programs so all program staff can benefit from promising practice and lessons learned
Sharing Evaluations Externally The evaluation officer considers how to position the foundation as a good citizen and leader by staying current with and contributing to the philanthropic evaluation field This is done in close coordination with the communications staff
OTHER DEPARTMENTS
The communications department helps program staff to consider with whom what how and when to share evaluation findings Particularly if informing the field is a key objective of sharing evaluation findings communications staff work with programs and consultants to plan for the best dissemination approach
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
9
Evaluation Principles and Practices Roles and Responsibilities
Program staff and communications staff are required to discuss proposed evaluations that raise specific legal concerns such as lobbying or election-related work with their legal team partner For example staff should speak with their legal team partner if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators plan to interview US or foreign legisla-tors The legal department should be contacted before requests for proposals are issued or evaluation work begins to ensure that the evaluation is compliant with the laws on lobbying and electioneering In addition the legal department will review contracts for all types of evaluation in accordance with the foundationrsquos normal contract review process
The Grants management staff is often asked to provide data from the grants management system This might include grant reports or other information relevant to a particular strategy The Hewlett Founda-tion will typically provide a consultant access to grant materials (proposals reports previous reviews etc) contact information for grantees and our own internal strategy materials
The finance department reviews spending on evaluations and works closely with the evaluation officer to track evaluation spending as a proportion of each programrsquos grant spending to determine whether it is in line with our benchmark of 15 to 2 percent of the total grantmaking budget14
EVALUATION CONSULTANTS
By definition our evaluations include third-party evaluators We expect the evaluators with whom we contract to follow the American Evaluation Association Guiding Principles for Evaluators15 We rely on the evaluators to gather data ensure appropriate confidentiality analyze and interpret findings and prepare and present findings to the different audiences identified
Typically we expect an evaluator to draw on a variety of quantitative and qualitative data collection techniquesmdashgoing beyond for example review of grant reports The data collection strategy should in most cases be geared toward capturing multiple and diverse perspectives and use varied sources to allow for triangulation across sources
We expect the evaluator will be dispassionate in reporting to usmdashie we want honest accurate and useful information and we do not expect or want flattery be transparent about time commitment and expectations for evaluation participants (program staff grantees other participants) collect all data with appropriate confidentiality as needed shareinteract with other evaluator(s) working with the same key partners synthesize data and provide findings in formats and ways that encourage feedback on interpretation and support discussion and learning
In reporting evaluators should address the evaluation questions and report on key successes and chal-lenges Evaluation reports should explain the evaluationrsquos limitations and include data collection proto-cols and tools in appendices In reports the evaluators should clearly distinguish findings conclusions and (if they are requested to provide them) a prioritized set of recommendations
Editorial guidance for evaluation reports to be shared publicly is included in Appendix B This editorial guidance is shared with evaluators with every evaluation contract
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
10
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Practice Guide Planning Implementation and UseThis practice guide follows the three stages of evaluation (a) planning (b) implementation and (c) practical use of the evaluation findings
PLANNING IMPLEMENTING USING
The seven evaluation principles apply across these three stages of an evaluationmdashand are visible at vari-ous points across the following practice guide
Included in the practice guide are case examples illustrating successes challenges and lessons learned
PlanningPlanning is perhaps the most important and complex part of evaluation We plan for evaluation at two levels
bull Big picture As part of strategy we use evaluative thinking to make explicit the assumptions that undergird our theories of change consider when and how evaluation data will be used and plan for commissioning specific evaluations to help us learn and adapt throughout the strategy lifecycle (see Appendix A)
bull Specific evaluation focus This includes articulating evaluation questions planning for the time and budget to engage grantees finding and contracting with an appropriate evaluation partner and being well prepared to use and share the findings
BEGINNING EVALUATION PLANNING EARLY
Even before commissioning a specific evaluation evaluative thinking can help sharpen a theory of changemdashan articulation of beliefs and assumptions explaining why proposed activities are expected to contribute to outcomes and long-term goals As part of the OFP process for example a program team should consider and make explicit its key assumptionsmdashoften the assumptions that link our theories of change together For instance consider this example of a simplified generic theory
bull If we invest in an innovative model we hope and plan for it to be successful andhellip
bull If proven successful it will be expanded to reach many more people
In between each link are potential assumptions to be tested
bull This innovative approach can be successful
bull Effective organizations exist that can implement this approach
bull This approach can become a ldquomodelrdquo and not just a one-off success
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
11
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
bull Others will be interested in adopting and supporting the model
bull Resources for growth and expansion exist to expand the model
bull When resources for growth are invested the approach can be widely and effectively implemented with high quality
As with many strategies each link builds on the one before it
Why Start Evaluation Planning Early
The Hewlett Foundationrsquos Organizational Effectiveness (OE) strategy which began in 2004 seeks to help nonprofits become high-performing organizations that are healthy sustainable and successful in achieving their goals OE provides relatively small targeted grants to help build Hewlett Foundation granteesrsquo internal capacity in areas like strategic planning board development and governance fundraising and communi-cations In 2014 the foundation commissioned its first-ever evaluation of the OE strategy A new OE officer had many questionsmdashincluding understanding what was working what was not why and whether OE grants were strengthening granteesrsquo health and resiliency in both the short and longer terms
The evaluation16 found that by and large OE grants deliver what they promise in the near term solid strategic plans fundraising campaigns leadership transition plans and so forth The evaluation also inspired new re-search into organizational assessment tools17 that might be useful for future assessment and evaluation But the evaluators were not able to address other important questions including whether OE support also pro-vides positive broader and longer-term value to grantees after the grant term Crucially the evaluators did not have the information they needed because the program had not planned for evaluation early enough and had not been collecting data consistently and systematically They may or may not have been able to answer vexing questions about broader and longer-term value but at least they would have been equipped to try
Starting evaluation planning early including sharing those plans with grantees and others who will likely be involved (eg other funders) protects against four common pitfalls (a) missing a ldquobase-linerdquo (b) not having data available or collected in a useful common format (c) surprised unhappy or unnecessarily burdened grantees and (d) a strategy or evaluation that is not optimally designed to generate the desired information to inform learning and decision making It also helps identify opportunities for small tests or experimentation in a strategy that could make a big difference for the strategyrsquos long-term trajectory
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
12
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Planning for evaluation does not mean casting those plans in concrete In fact given that our strategies typically unfold dynamically it is essential to revisit and modify evaluation plans as a strategy matures and refreshes
Purpose-Driven Evaluation Will Shift Focus Over Time
The Madison Initiative18 which focuses on strengthening US democracy and its institutionsmdashespecially Congressmdashin a time of political polarization commissioned the Center for Evaluation Innovation to work closely with foundation staff throughout the initiativersquos initial three-year exploratory period During these early years the Madison Initiative team selected a developmental evaluation design an approach that em-beds evaluation in the process of strategy development and implementation The role of developmental evaluators is to be a ldquocritical friendrdquo to the strategy team asking tough evaluative questions uncovering assumptions applying evaluation logic and collecting and interpreting evaluative data to support strategy development with timely feedback
Early in the evaluation the evaluators developed a systems map This map helped the Madison team estab-lish a common understanding of what the initiative was trying to domdashand the key variables both inside and outside of the Madison Initiativersquos funding areas Working with the map helped the team recast its thinking and see holes and gaps in different ways which then allowed the team to change the grantmaking avenues it pursued However the map was less helpful for informing decisions specific to what the foundation could do relevant to their grant clusters
Thus as the strategy matured smaller evaluations were commissioned of specific grant clusters working toward similar outcomes These cluster evaluations informed strategic pivots and grantmaking decisions As an example by early 2016 the Madison Initiative had been investing in campaign finance grantees for several years and began wrestling with whether in light of technological advances and talk of ldquothe big data revolutionrdquo foundation support for basic campaign finance data collection and curation was still necessary Findings provided by the evaluator affirmed the teamrsquos convictions that these grantees in fact do play an important role in reform a decision was made to support large long-term funding to those grantees
After a strategy refresh in 2016 the team continued its focus on smaller targeted evaluations of grantee clusters working toward specific outcomes A series of sequenced evaluations will examine questions about what is and is not working These evaluations are timed to produce findings at key junctures in the grantmaking process in order to position the Madison Initiative and its grantees to act on lessons learned
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CHOOSING WHAT TO EVALUATE THROUGHOUT THE STRATEGY LIFECYCLE
We cannot (and need not) evaluate every aspect of a strategy nor do we evaluate every grant One crite-rion for what program staff choose to evaluate is their openness to changemdashand readiness to challenge strongly held beliefs Often the most strongly held beliefs are those assumptions embedded in the theo-ry of changemdashthat an innovation will succeed for instance or that others will adopt a ldquosuccessful modelrdquo
Several other criteria guide our decisions about where to put evaluation dollars Highest priority is given to the following considerations
bull Urgency for timely course correction or decisions about future funding
bull Opportunity for learning especially for unproven approaches
bull Risk to strategy reputation or execution
bull Size of grant portfolio (as a proxy for importance)
Program officers with the supervision of program directors determine what aspects of the strategy to evaluate and when Teams submit an evaluation plan on an annual basis and update these plans each year
Most of the time program staff will plan for an evaluation to focus on a strategy substrategy or cluster of grants that meet these criteria rather than focusing on a single grant The exception is when a grant is essentially operating as an initiative or cluster in and of itself
OUTCOME-FOCUSED PHILANTHROPY STRATEGY HIERARCHY
PROGRAM
STRATEGY OR INITIATIVE
SUBSTRATEGY
GRANT CLUSTER
INDIVIDUAL GRANT
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
14
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
While we encourage choosing strategically what to evaluate we rec-ommend that all strategies should begin an evaluation (in whole or in part) within three years of origination or refresh Planning for an evaluation within that time frame encourages us not to let too much time go by without an external set of eyes on progress doing so also sets us up with evaluations that can inform strategy refreshesmdashwhich most strate-gies go through roughly every five years
Most strategies operate with several substrategies often with multiple clusters nested in each Many program staff identify smaller substrat-egy and grant cluster areas as highest priority for evaluation to inform strategy and grantmaking decisions For example the Cyber Initiative chose to evaluate its work on network building19 early in the life of the strategy since that work was identified as a necessary element to support the overall strategy goal After a strategy refresh the Glob-al Development and Population Programrsquos International Reproductive Health team identified for evaluation several substrategies it was intro-ducing These included testing new tools and approaches20 working in Francophone West Africa21 and supporting local advocacy in sub-Saharan Africa22 These substrategies were identified because they held more risk for successful implementation and because the team believed there were benefits to early learning and adaptation In fact that plan turned out perfectly as the specific evaluations for these substrategies did in fact substantially inform decisions during implementation
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track Such a compre-hensive assessmentmdashmost commonly used to inform a strategy refreshmdashlikely requires synthesis of multiple evaluations of smaller cluster-level evaluations summary and analysis of relevant implementation markers and active reflection on and interpretation of the results in context This process can be more of a ldquoquilting artrdquo than an exact science There is value in having a third party assist with such an evaluation to increase ob-jectivity The 2018 Western Conservation strategy refresh23 for example relied on a third-party consultant to weave together a comprehensive ret-rospective evaluation24 a set of cluster-specific evaluations a deep dive into equity diversity and inclusion issues among grantees and research on best practices for effectively communicating and collaborating with rural Western communities
Frequently the foundation uses regranting intermediaries to extend its reach and increase the impact of its grant dollars and results Because we are delegating to these intermediaries what might be considered our stewardship role we have an even greater responsibility to evaluate their efforts By definition large intermediaries rank high on the risk and size criteria above and evaluating them typically offers important learn-ing opportunities Also whenever we contribute to creating a new inter-
When it comes to strategy-level evaluation typically no single evaluation can tell us if a strategy has been successful or is on track
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
15
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
mediary organization or fund the launch of a major new initiative it is important to evaluate not only the strategic elements but also issues of organizational health (eg leadership and board development) and effective execution (eg to what extent is something happening as planned)mdashchallenges that vex many startups In some cases we commission an evaluation ourselves such as the evaluation of the Open Educational Resources Research Hub25 in other cases particularly for funder collaborations we may contribute to pooled funding for evaluations This was the case with the evaluation of the Fund for Shared Insight26 where many funders contribute and the intermediaries commission the evaluation When we are interested but will not be involved in a decision-making role we might give a supplemen-tal grant to a grantee for them to commission an evaluation and serve for example on an evaluation advisory committee As with all of our evaluations it is important to be clear on the purpose and to weigh the benefits and challenges of each approachmdashwith regard to quality buy-in likelihood of use and broad sharing
Multiple Evaluations Across the Strategy Lifecycle Strengthen Learning and Adaptation
When the Hewlett Foundation introduced its Deeper Learning strategy in 2010 the goal was to spread a con-crete set of skills that American students should be learning The strategy anticipated that high schoolers who gain academic knowledge alongside inter- and intrapersonal skills will be better prepared as college students workers and citizens The Deeper Learning team commissioned multiple evaluations over its first six years
The team commissioned a strategy-level evaluation27 in 2013 with a formative purpose to assess the execu-tion of the strategy during its first four years assess the viability of the strategyrsquos assumptions and provide concrete recommendations on how to improve the prospects of attaining the ultimate 2017 goal to ensure that 8 million students (about 15 percent of the K-12 public school population) were taught higher-order skills
In deciding which aspects of the strategy to evaluate over the years following the 2013 evaluation the team continued to lead with purpose and looked at which key decisions could benefit most from evaluation The team also considered how smaller evaluations could serve as critical input to inform a larger strategy-level summative evaluation which would likely be commissioned in 2016
Between 2014 to 2016 the team commissioned numerous cluster evaluations One evaluation was helpful in informing shifts in grantmaking when program staff needed to reduce (and more strategically focus) its fund-ing of policy work in order to increase funding for other aspects of its strategymdasha recommendation that came out of the 2013 formative assessment Staff used evaluation findings in a number of ways including to iden-tify the grantees best positioned to successfully advance Deeper Learning-aligned policiesmdashthose with both strong capacity for policy impact and high alignment with Deeper Learning goals the team shifted funding for these organizations toward longer larger and more general operating support grants Another evaluationmdashof communications effortsmdashwas useful for establishing the (then) current status of how Deeper Learning was communicated and understood as a term and focus in the field and granteesrsquo roles in shaping it
As the time approached for their planned summative evaluation28 of the strategy the team settled on a set of broader strategy-level evaluation questions with the intentional purpose of synthesizing progress from 2010 to 2015 As the team described it ldquowhile the substrategy evaluations were precisely designed to test the in-dividual links in our logic model the broader 2016 summative evaluation would look at what happened in the boxes and interrogate the assumptions about the arrows linking the boxes togetherrdquo This summative evalua-tion (along with a host of other inputs) then informed the programrsquos strategy refresh which began in late 2017
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
16
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It is also important to plan for commissioning an evaluation in cases where the foundation exitsmdashto generate lessons that will be useful to multiple stakeholders inside and potentially outside the foundation The Nuclear Security Initiative summative evaluation is a good example of this29 The evaluators summed up the lessons learned from this seven-year initiative with respect to successes and challenges and how well the Initiative handled the exit The evaluation also provided a broader set of lessons on how the Hewlett Foundation and other funders might track progress and evaluate policy-re-lated efforts30 Many of the lessons learned from the Nuclear Security Initiative evaluation are incorpo-rated into OFP guidance on preparing for and handling an exit
Engaging Grantees is Critical to Building Trust and Buy-In
In 2014 the Global Development and Population programrsquos International Reproductive Health strategy began funding new approaches to increase the effectiveness of service delivery by pairing service delivery grantees with organizations that could bring new insights about service designmdashdrawing from the fields of behavioral economics and human-centered design (HCD) As the strategy began program staff commissioned an eval-uation to understand whether and how this new effort was working
The program officer took several steps to ensure the success of the evaluation including developing an RFP being clear on the evaluationrsquos purpose identifying a set of evaluative questionsmdashand sharing these items with some grantees for input But early into the implementation of the evaluation there were rumblings about how the evaluation was being carried out Grantees wondered ldquoWhy are these evaluation questions being askedrdquo ldquoWhy are the evaluators taking the approach to data collection they are usingrdquo ldquoAre there important ways in which the evaluators are biased toward their own approach to HCDrdquo These rumblings disrupted the ability of the evaluators to effectively carry out an evaluation
It became clear that these evaluators would not be able to build trust and gain enough confidence to gather the data needed for the evaluation As a result after the completion of the contract for an initial evaluation design and inception phase the program officer decided to end that evaluation relationship Yet evaluating the work remained important So the program officer tried againmdashthis time doing even more to get input and buy-in from all the grantees who would be involved in the evaluation To do so the program team took several steps First they traveled to and met with representatives from each of the grantees involved in the effort and asked what questions they wanted answered and what they would be looking for in an evaluation Second based on what the program officer heard she put together a scoping document that identified overall pur-pose (key audiences and intended use) along with which questions a Hewlett Foundation-commissioned evaluation would answer and which might be addressed by other funders or if appropriate by individual grantees to answer as part of their own evaluation Third when she received evaluatorsrsquo proposals from a subsequent RFP process she ran the finalists by the grantees to hear of any concerns before finalizing the selection Finally she developed an advisory committee composed of representatives from each grantee and other funders and requested that the group weigh in at key junctures of the evaluation including giving input on the design and data collection strategy getting feedback on initial evaluator findings and finally discussing findings and recommendations31 at a ldquoco-creation workshoprdquo
Taking the time to get grantee input early and often and using an advisory group as a mechanism for grantee engagement throughout the evaluation process helped build trust and limit surprises An advisory committee composed of grantee representatives and other funders can support the use of findings for learning and project improvement
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
17
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ENGAGING WITH GRANTEES
Communication with grantees early and often about expectations for evaluation is crucialmdashthough what information is communicated and how that information is communicated will depend on the purpose of the evaluation commissioned and the granteersquos role in it Often this expectation needs to be communicated and reinforced several timesmdashat a grantrsquos inception again as a specific evaluation is planned during implementation of evaluation activities and data collection and during the use and sharing phases For example at a grantrsquos inception program staff need to inform grantees that they may be expected to participate in an evaluation share data with the foundation and evaluators and have the results shared publicly in some formmdashfull executive summary or presentation The shared agreement from this discussion is then reflected in the language in the Grant Agreement Letter As one grantee who reviewed an early draft of this guide advised us ldquoDonrsquot sugarcoat what the evaluation experience will entailrdquo In the long run everyone does better when expectations are clear
Some evaluations involve large numbers of grantees (for example strategy-level evaluations can include the work of dozens to hundreds of grantees) but even in these cases to the extent feasible it is import-ant to get at least some input from grantees who will be most directly involved in an evaluation
Be Good Clients
An effective consulting engagementmdashwhether for an evaluation or anything elsemdashrequires that we be en-gaged and thoughtful clients For evaluation this requires planning for and allocating enough time to be a full participant in the process planning data collection analysis and interpretation and use and sharing So what should you do
1 Allocate enough time for you to participate in the evaluation as needed Depending on the type of evalua-tion you commission this tends to be more necessary at the beginning and toward the end of an evaluation process An evaluation where you want interim results or a more participatory approach will take even more time for you and the evaluator
2 Check in with the evaluator about the time commitment they need from you and in advance define a process to ensure the evaluators receive the support and information they need to do a great job
3 At the start of the evaluation take the time to orient consultants to the foundationrsquos values and process-es Team culture values and dynamics are important and it will help the consultant to understand what these are and what issues the team might be grappling with
4 Give evaluators the time needed to design gather data analyze reflect and interpret Donrsquot drag your feet in commissioning an evaluation and then squeeze the evaluators with an unrealistic time frame
5 Make sure evaluators are aware of and you and they have the time and budget to address priorities related to grantee engagement and DEI issues including as relevant the questions posed the methods used and the process for interpreting and using the findings (Appendix C and the Equitable Evaluation site offer helpful resources for this step)
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
18
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ALLOCATING SUFFICIENT TIME
Planning for and allocating sufficient timemdashfor program staff and the evaluatormdashacross the phases of an evaluationrsquos life is a key ingredient for an evaluation that is useful and used In general program officers are expected to effectively manage one significant evaluation at any given time this in-cludes proper oversight and engagement at each stage from planning through use and sharing of results
Though evaluations take time they should be considered an important part of a program teamrsquos work at each stage of the strategy lifecycle interacting with grantees and others involved learning what is working and what is not and informing course corrections Program staff who have engaged in this way with evaluations have indicated the time spent has paid off
In general program officersmdashwho take the lead on the evaluations they commissionmdashwill spend 5 to 20 percent of their time planning managing and determining how to use the results from evaluations This overall expectation is amortized over the course of each year though there are peri-ods when the time demands will be more or less intensive The most intensive time demands tend to occur at the beginning and end of an evaluationmdashthat is when staff are planning and then using results During these periods full days can be devoted to the evaluation For instance planning requires con-siderable time to clarify purpose refine evaluation questions pull together the necessary documents for the evaluation engage grantees choose consultants and set up contracts Program associates also typically spend quite a bit of time during these phases related to contracting document collection and sharing and convening activities
During the use phase (including sharing) the time demand ramps up again as staff spend time meeting with consultants interpreting results reviewing report drafts checking in with grantees and others communicating good or bad news identifying implications for practice and sharing findings internally and externallymdashincluding publicly Typically the time demand for the program staff is not as intensive while the evaluator is implementing the evaluation data collection activities and doing the analysismdashthough program staff should not underestimate the time it will take to stay in touch ask questions of the evaluators and the grantees discuss draft protocols and ensure everything is proceeding well
THE TIME REQUIRED BY PROGRAM STAFF VARIES OVER THE COURSE OF AN EVALUATION SOME EVALUATIONS LIKE DEVELOPMENTAL EVALUATIONS MAY REPEAT THIS CYCLE
TIM
E R
EQU
IRED
PLANNING IMPLEMENTATION USING AND SHARING
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
19
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
CLARIFYING AN EVALUATIONrsquoS PURPOSE
The purpose of an evaluationmdashwhich includes its planned audience use and timeline for when evaluation findings will be most usefulmdashis cen-tral Questions methods and timing all flow from a clear understanding of how the findings will be used by whom and when
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
Below we note three main uses (not intended to be mutually exclusive) for our evaluations
bull To inform foundation practices and decisions Evaluations with this aim may inform our decision making about funding or adapting an overall strategy or substrategy setting new priorities or setting new targets for results These evaluations are typically designed to test our assumptions about approaches for achieving desired results While we share these publicly in keeping with our principles around openness and transparency the primary audience for these evalua-tions is internal foundation staff
bull To inform granteesrsquo practices and decisions At times the founda-tion may want to fund or commission evaluations that can be used by grantees to improve their practices and boost their performance When the interests of the foundation and grantees overlap it can be worthwhile to commission evaluations designed to be of value to both Collaborating in this way can promote more candor and buy-in for the ways data are collected and results are used In these cases it is worthwhile to consider ahead of time the value of having an eval-uator prepare an overall public report as well as individual private reports for each or select grantees This costs more but there is also typically greater benefit to the individual organizations
bull To inform a field Evaluations that include informing a field or other funders as a distinctive purpose should be intentional about involv-ing others (such as peer funders or others who we hope will also benefit from the evaluation) in considering what questions would be most valuable to address These evaluations are typically designed with influence in mind so that others can learn what we are learning and help shape field-building or build trust in an idea or approach Although all our evaluations involve sharing publicly when inform-ing a field is identified as an important purpose it is worthwhile to work ahead of time with the evaluator to determine whether it would also be helpful to consider additional support for preparing fi-nal products for dissemination (eg from communications experts editors or data visualization specialists)
From the very beginning of the evaluation process it is important to plan how the results will be used if in the beginning you take time to imagine how you might respond to different results scenarios you are halfway toward actually using what you find out
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
20
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Most of the evaluations we are covering in this guidance have the dual purpose of informing foundation decisions and practices and those of our granteesmdashin both cases to support ongoing learning adjustment and improvement
We Distinguish the Evaluations We Commission from the Research We Fund
There is a lot written on the differences between evaluation and research32 Almost every program funds research as part of a strategy itself to identify new opportunities and best practices in an area to identify gaps in a landscape or to generate knowledge for a fieldmdashand to have that knowledge shape policy and practice For example the Madison Initiative funds research on digital disinformation the Knowledge for Better Philanthropy strategy funds research on best practice in philanthropy and the Global Development and Population Programrsquos Evidence Informed Policymaking substrategy funds organizations committed to commissioning impact studies
The research studies funded are typically distinctmdashin terms of purpose process and audiencemdashfrom the evaluations we commission to understand to what extent for whom how and why a strategy is working or not Indeed because these research studies are part of our strategies they are often subjects of the evaluations that we commission For example in the evaluations of the Knowledge Creation and Dissemination33 and the Population and Poverty Research34 strategies program staff addressed evaluation questions about the quality and reach of the research and adoption by intended audiences
STRATEGY
Knowledge for Better Philanthropy
In addition to supporting basic and applied re-search on philanthropy grants supported leading journals in the field that disseminate knowledge and efforts to create new systems and platforms that would provide better solutions to learning and professional development
bull Stanford Social Innovation Reviewbull Center for Effective Philanthropybull Bridgespan Groupbull Issue Labbull National Committee for Responsive Philanthropy
bull How do grantees measure and understand their impact to-date related to knowledge production and dissemination aimed at informing influencing and improving donorsrsquograntmakersrsquofundersrsquo thinking and decisionmaking
bull What knowledge on philanthropy and other aspects of the social sector is being produced by grantees
bull How have grantees disseminated knowledge on philanthropy and other aspetcs of the social sector
bull Who is using the disseminated knowledge and how is it being used
bull To what extent did PopPov strengthen the field of economic demography
bull What contribution has PopPov made to the evidence base
bull To what extent did PopPov yield policy-relevant research
bull How did the design and implementation of PopPov affect outcomes
Between 2005-2014 the Hewlett Foundation in-vested more than $25 million in a body of research on the relationship beetween population dynamics and micro- and macroeconomic outcomes
bull Center for Global Developmentbull Population Reference Bureaubull World Bank
Population and Poverty Research Initiative (PopPov)
RESEARCH (PART OF STRATEGY) EVALUATION QUESTIONS
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
21
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
3-4 MONTHS 3-4 MONTHS 3-4 MONTHS 3-4 MONTHS
Write RFP Find Evaluator and Contract
Discuss and Interpret Findings
KEY JUNCTURE
Apply the Findings in
Practice
Sharing and Follow-Up
Design and Data Collection
When considering use it is important that we examine our level of openness to a range of results and pin down what degree of rigor and strength of evidence in the evaluation would be required to change the minds of the various users of the evaluation Evaluation is worthwhile only if one can imagine being influenced by the findings What strength of evidence will change our minds and the minds of the others we hope to engage If we are not willing to change our minds or the degree of evidence to change our minds is too costly or time-consuming to obtain we should reconsider the value of spending money on evaluation
What is the timeline for when the findings will be most useful One criticism of evaluation is that results often come too late to be useful But that is in our control There are trade-offs to keep in mind but it is important not to sacrifice relevance by having evaluation findings be delivered too late to matter Of course considering evaluation early as part of strategy development will help define when specific information will be needed
Consider the following set of questions in preparing a timeline for evaluationmdashone that includes important dates decisions and a cushion for inevitable lags If we want to inform foundation decisions what is our timetable for receiving at least preliminary results How rigid is that timetable Backing up from there when would we need to have results in order to make sense of them and to bring them forward for funding considerations Backing up again how much time do we need to find the right evaluator and give the evaluator enough time to design an effective evaluation then gather analyze synthesize and bring those results to us If we want actionable information it is essential to grapple with what is knowable in what time frame If we are aiming to inform grantees how might their budgets or program planning cycles affect the evaluation timetable Grantees also need time to make sense of findings and act upon them If our purpose is to inform the field or to engage other potential funders in an area are there seminal meetings or conversations that we want an evaluation to influence Are there planning processes or budget cycles that might be important to consider in our evaluation planning
Be sure to consider how others in the foundationmdashprogram peers the evaluation officer communica-tions staff and at certain points grants management and legalmdashwould also be helpful or necessary For example if evaluation questions refer to legislation ballot initiatives or campaigns and elections or if evaluators will interview US or foreign legislators you must talk with legal before getting started Leave a couple of weeks for contracting and contract review
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
22
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
DEFINING EVALUATION QUESTIONS
Our evaluations begin with and are guided by clear crisp questions Crafting a short list of precise evaluative questions increases the odds of receiving helpful answersmdashand a useful evaluation It also increases the odds that evaluation will support learning because the program officer (and grantees) can ask questions that will be most informative for their learning Well-designed questions can not only clarify the expected results but also surface assumptions about design causality time frame for results and data collection possibilities These surfaced assumptions and questions can then help sharpen a theory of change and ensure effective planning for evaluation and learning
Unfortunately many evaluations begin to go awry when questions are drafted It is useful to start by distinguishing between the following areas of inquiry Although not every evaluation should seek to answer this full range of questions the categories below offer suggestions for effective investiga-tion depending on the nature and maturity of the strategy or area of interest selected for evalua-tion These are general examples of questions and should be tailored to be more precise
bull Implementation How well did we and our grantees execute on our respective responsibilities What factors contributed to the quality of implementation In much of the social sector evidence shows that most program strategies and program interventions fail in execution This makes evaluating implementation very important for driving improvement understanding the ingredients of a successful or failed approach and replicating or adapting approaches over time
bull Outcomes What changes have occurred and why If not why not How do the changes compare with what we expected and the assumptions we made To what extent and why are some people and places exhibiting more or less change To what extent is the relationship between implemen-tation and outcomes what was expected To be able to answer these questions it is enormously helpful to have planned an evaluation from the outset so that measurements are in place and changes are tracked over time While we tend to use implementation markers to track progress toward outcomes we use evalu-ation to analyze more systematically underlying issues related to what caused or contributed to changes in these outcomes why why not and for whom
bull Impact What are the longer-term sustainable changes To what extent can these changes be attributed to the funded work or could the work be determined to have contributed to these im-pacts Impact questions are typically the most complex and costly to answer particularly for much of the field-building systems-level and research- and advocacy-related work that we fund
bull Context How is the (political policy funding country) landscape changing Have changes in the world around us played an enabling or inhibiting role in the ability to effect change Often our theories of change involve assumptions about how the world around us will behave and unanticipated eventsmdashconflicts new governments social protests disease technological or scientific breakthroughsmdash can accelerate or slow progress toward long-term goals Understanding these interplays can help us avoid false conclusions
bull Overall Strategy and Theory of Change Did our basic assumptions turn out to be true and is change happening in the way we expected In order to answer questions about the overall strategy it is likely that you will draw from more than one evaluationmdashwith findings synthesized in order to gain deeper insight It is helpful to develop overarching evaluation questions early on in the strategy process however to ensure that you are gathering the pertinent information you may need from each
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
23
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
What can you do to make these general evaluation questions more precise and evaluative
bull Make more specific (eg be more specific about what is meant by a word or phrase)
bull Tie more closely to intended use (eg add a note about why answering this question will be helpful and for whom)
bull Make more realistic (eg add time frame location narrow scope)
bull Add ldquocompared tordquo as part of the question (eg compared to other approaches compared to where we expected)
bull Ask to what extent whywhy not How For whom (rather than just ldquodid x happenrdquo)
bull Special Case Evaluating a Regranting Intermediary This can require an additional set of questions How and to what extent is the intermediary adding value to its grantees Is it just a go-between supporting the transaction of regranting funds without adding significant additional value Or is the intermediary able to offer important technical assistance to organizations by virtue of being closer to the ground Where and for whom is the intermediary adding the most value and where is it adding the least What are the enablers and inhibitors to an intermediaryrsquos high perfor-mance How does this intermediaryrsquos performance compare to other intermediaries How trans-parent efficient and well managed is the subgranting process and related communications and what is the subgranteesrsquo experience Did they get clear communications from the intermediary or support to figure out how to prepare budgets that reflected their full costs
bull DEI Issues When we consider issues of diversity equity and inclusion in our strategies we should also consider how DEI shows up in our evaluation questions Include questions to under-stand the perspectives and insights of those whose voices are least heard As a hypothetical exam-ple a gender-blind evaluation may ask To what extent were the goals of the program met Why or why not A gender-inclusive evaluation may instead ask To what extent were the goals of the program metmdashand how did the effects differ among males females and others When changing systems that drive inequity is part of the strategy then the evaluation might ask questions about whether and how those systems have changed why why not and for whom At the very least include questions about whether there were any unintended consequencesmdashand for whom
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
24
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Integrating Diversity Equity and Inclusion into the 2018 Western Conservation Strategy Evaluation and Refresh
The Western Conservation strategyrsquos long-term goal is the preservation of biodiversity and the conserva-tion of the ecological integrity of the North American West for wildlife and people In preparing for its most recent strategy refresh program staff commissioned evaluations to assess the strategyrsquos short-term time-bound initiatives such as California Drought and Canadian Boreal Forest as well as an evaluation of the overall strategy35
Among other evaluation questions about progress successes and challenges the team included ques-tions related to issues of diversity equity and inclusion The team sought an evaluation that would (a) unpack the foundationrsquos blind spots related to the diversity of the current Western Conservation grantee portfolio (b) identify the relationship of these DEI values to the strategyrsquos policy objectives and (c) rec-ommend how the Hewlett Foundation could be more deliberate about supporting grantees led by and serving communities of color and those working to make greater progress by embracing the values of equity and inclusion within their organizations and in how they approach their work in the world
The strategy-level evaluation paid careful attention to soliciting diverse perspectives For example the evaluators talked to Hewlett Foundation grantees farmers and ranchers tribal governments sportsmen and women Latino and African-American organizations faith communities and youth and included their direct feedback along with other qualitative and quantitative data Paying specific attention to whom they were hearing frommdashwith foresight time and intentionalitymdashproved valuable for providing new insights
Perhaps most important among the many lessons from this intensive evaluation process was new under-standing of the critical role of inclusivity in securing lasting conservation outcomes One ah-ha moment for the team from the evaluation Equity and inclusion efforts must be paramount in the grantmaking strategy and precede a diversity push in order to intentionally signal to the field their importance and to avoid tokenism in hiring and outreach strategies (which might come from a focus on diversity alone) The evaluation findings also reaffirmed the value of diversity equity and inclusion as ldquonot simply a moral issue but a policy-making imperativerdquo Building on these findings the new strategy argues that to endure the winds of political change conservation solutions must be place-based and account for the diverse cul-tural economic social and ecological needs of a community which requires engaging a broader range of stakeholders and constituencies in the development and defense of conservation solutions Today the Western Conservation grantmaking portfolio and strategy36 reflect a vision of a more inclusive conserva-tion movement for the long-term benefit of western communities ecosystems and wildlife
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
25
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
HYPOTHETICAL lsquoTeacher as Learnerrsquo Initiative Lessons on Crafting Precise Evaluation Questions
Imagine that we are supporting a new initiative called ldquoTeacher as Learnerrdquo that aims to improve the quality of teaching and learning for students of all backgrounds in different places via a network of 100 self-organized groups called ldquocommunities of practicerdquo Each group of local teachers is professionally facilitated and focused on their specific capacity needs Having organized themselves around issues of local importance the communities of practice draw on regional resources as needed The initiativersquos key assumption based on some evidence in other fields is that a blend of professional support and local ownership will lead to improved outcomes If this approach seems successful after an initial period of innovation we might develop an experiment to rigorously assess impact
What are our evaluation questions
POOR SAMPLE QUESTION
Was the Teacher as Learner theory of change successful
This question has limited value for several reasons First it is vague Usually a theory of change has mul-tiple dimensions and contains many assumptions about how change will happen A useful evaluation question is explicit about which interventions and assumptions it is exploring or interrogating A vague question gives the evaluator too much discretion This often sets us up for potential disappointment with the findings when we receive an evaluation re-port that is not useful and does not answer questions of importance to us
A second related point it is unclear whether the question is aimed at issues of execution (eg Did x happen) or issues related to the ldquocausal chainrdquo of events (eg If x happened did it catalyze y) It is often useful in an evaluation to look at execution and outcomes with a distinct focus as well as the relationship between them
Third the definition of success is unclear allow-ing the evaluator too much discretion Does suc-cess mean that 80 percent of what we hoped for happened What if 60 percent happened What if two out of three components progressed exactly as planned but a third delayed by an unforeseen per-sonnel challenge has not yet been implemented Asking a dichotomous YesNo question about an un-specified notion of ldquosuccessrdquo will be less helpful than a few focused questions that precisely probe what we want to learn and anticipate how we might use the answers
GOOD SAMPLE QUESTIONS
About implementation
1 How and to what extent did the Teacher as Learn-er initiative create a network of local self-orga-nized communities of practice
2 What was the nature of the variation in areas of focus for the communities of practice
About intermediate outcomes
3 To what extent did teachers adopt or adapt improved teaching methods after participating in the communities of practice
4 What were the key factors that enabled or inhibited teachers from adopting new teaching methods
About outcomes
5 In what ways and by how much did these teachersrsquo students improve their learning
6 Is there any variation in studentsrsquo learning gainsmdashfor example by gender or by students with disability If so what are possible explanations for that variation (including student teacher or community characteristics and features and ap-proaches used in the communities of practice)
Why are these better questions As a valuable be-ginning they break one vague question about success into clear specific ones that generate insight about dif-ferent steps in the initiativersquos causal chain which parts may be working well and as expected which less well and possible explanations for this They give more di-rection to the evaluator about our specific areas of in-terest And although they still need to be elaborated on with specific measurement indicators and meth-ods of data collection they are designed to generate data that can be used to correct course
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
26
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
IDENTIFYING METHODS
Most strong evaluations use multiple methods to collect and analyze data The process of triangulation allows one methodrsquos strengths to complement the weaknesses of another For example randomized exper-iments can determine whether a certain outcome can be attributed to an intervention but complementary qualitative methods are also needed to answer questions about how and why an intervention did or didnrsquot workmdashquestions that are central to replication or expansion
Multiple methods help reduce bias as does active consideration of how the methods are applied For instance if an evaluation is primarily based on qualitative key informant interviews it will be important to include re-spondents who are not cheerleaders but may offer constructive critiques
The evaluator should be primarily responsible for identifying the meth-ods but it is helpful to set expectations that the evaluation will include multiple methods and capture diverse perspectives and that it will use both qualitative and quantitative data collection
Grantees are good sources regarding methods Once an evaluator is selected the evaluator should review data collection procedures and protocols with (at least some) grantees in order to assure consistency and applicability Sometimes grantees might give input on methods for example if they are aware that a certain methodology would prove inef-fective or the language in an interview or survey might need adjustment
Our goal is to maximize rigor without compromising relevance While most evaluations of our strategies cannot definitively attribute impact to the funded work the essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of inter-ventions organizations populations or regions Even when there is no formal counterfactual (ie example of what happened or would have happened without the strategy) it can be helpful to engage in discussion of what else might be happening to explain findings in order to challenge easy interpretations of data and consider alternative explanations
Evaluators should be expected to take a systematic approach to causal inference At the very least this includes checking that the evidence is consistent with the theory of change and identifying alternative explana-tions to see if they can be ruled out as the cause of any observable results Given the nature and complexity of many Hewlett Foundation strate-gies it is likely that evaluators will need to identify noncounterfactual evaluation approaches that go beyond simple comparison For example contribution analysis37 process tracing38 qualitative comparative analy-sis39 and QuiP40 are techniques that have been developed to do this It is not necessary for program staff to know the details of these approaches but it can be helpful to surface in discussions with potential evaluators why they are suggesting a specific approach A good resource for learning about different types of evaluation is BetterEvaluationorg
The essence of good evaluation involves some comparisonmdashagainst expectations over time and across types of interventions organizations populations or regions
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
27
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
As noted in the section on purpose it is worth checking with intended users (including yourself as commissioner) to understand what degree of rigor is necessary for the findings to be useful for those who are in the position to use and make changes based on the findings An evaluation is worth doing only to the extent that it is going to affect decisions or challenge beliefs In some cases this means the strength of evidence needed will be time-consuming and costly to obtain In other cases the users might agree that less evidence is necessary to inform course correction or decision making
Be prepared to discuss with the evaluator how the data will be gathered analyzed and shared with regard to confidentiality Will the data be kept confidential with no names or unique identifiers attached Will grantee organizations be identified There are pros and cons to different types of data gathering If an evaluator tells the respondent the information gathered will be kept confidential they cannot then turn around and share the name of the grantee or others who responded a specific way If the information is only going to be useful with identifiers attached then the pros of gathering it that way may outweigh the potential cons of too much courtesy bias
CRAFTING AN RFP FOR AN EVALUATOR
Once you have identified the purpose and an initial set of evaluation questions it is time to identify a third-party evaluator Start by crafting a thorough request for proposal (RFP) Evaluations chosen through competitive selection processesmdasheven if only involving conversations with two or three differ-ent evaluators rather than a formal paper proposal processmdashtend to offer greater opportunity to find an evaluator that will make the best partner for the evaluation project At a minimum if an evaluator is chosen without an RFP (perhaps in cases where the evaluatorrsquos work is already well known to the person commissioning the evaluation or the evaluation is a continuation of earlier evaluation work) create an evaluation purpose document for discussion with the evaluator Establishing clarity about
Increasing Rigor and Relevance
In 2015 the Performing Arts programmdashwhich aims to sustain artistic expression and encourage public en-gagement in the arts in the Bay Area--commissioned a midpoint evaluation of their strategy41
Two key questions in commissioning the evaluation were which geographic and demographic communities have benefitted from Hewlett Foundation support and where are the gaps Data from an audience research project the program had previously supported for a group of granteesmdashthe Audience Research Collaborative (ARC)mdashproved very informative for addressing this question The evaluators were able to compare the de-mographics of the audiences of the grantees to the broader demographics using census data for the area As a result the Performing Arts program could see where they had strengths and weaknesses and where they could diversify their portfolio to better reach the participants and artists they hoped to reach Since they had anticipated early on that demographic data would be valuable they were able to build in a comparison point as part of their evaluation
Itrsquos important to note that the data collection strategy was not without its limitationsmdashwhich is the case with all evaluations For example the survey methods used may have introduced bias towards more white female and highly-educated respondents Whatrsquos more US Census categories themselves have not kept pace with rapid demographic change and donrsquot reflect the true diversity of our society something many participants in ARC addressed by collecting data about both the census categories and expanded categories that better reflect the communities they serve (for gender identity for example) Nevertheless the findings were valuable for the program and the planning and foresight paid offmdashand they went in with eyes open to the limitations
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
28
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
the expectations for the project (on all sides) helps to make sure that all parties are in agreement regarding the purpose approach roles and time commitments
The basic elements of an RFP to engage an evaluator include back-ground information about the strategy substrategy cluster or grantee background information about the evaluation its purpose intended audiences and anticipated use key evaluation questions known available data sources time frame for receiving results preferred deadline for the deliverables and types of deliverables required (including internal foun-dation external grantee field and public-facing deliverables) evaluator qualifications and rolesexpectations and evaluation budget (amount of available funding)
Include language in the RFPs and ultimately the contract scope of work so that the evaluator is aware of the foundationrsquos expectation that there will be a product that will be shared publicly
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants (See Appendix C) Some program staff have engaged grant-ees in this part of the process by circulating draft scoping documents that summarize purpose key evaluation questions and proposed timelines for data collection synthesis and reporting Grantees will likely offer useful feedback on opportunities or challenges for timing the collection of data
CONSIDERING WHETHER OR NOTmdashAND HOWmdashTO HAVE THE EVALUATOR OFFER RECOMMENDATIONS
One important area to be clear on before beginning an evaluation is whether you want the evaluator to include recommendations along with the findings Sometimes asking an evaluator not to provide recom-mendations works best since the evaluator may not be in a position to truly understand the context within which program staff will be making strategic decisions In fact we have found that recommenda-tions can sometimes be counterproductive because if they are off-base or naive they can undermine the credibility of the overall evaluation
CHOOSING AN EVALUATOR AND DEVELOPING AN AGREEMENT
The ideal evaluator is strong technically (typically in social science research techniques) has subject matter expertise is pragmatic and communicates well both verbally and in writingmdashwhether about good or bad news Cultural awareness and sensitivity to the context in which nonprofits are operating are also very important as is the ability to work well with grantees and to find ways to communicate results in ways that are useful If we cannot find that full package it is sometimes appropriate to broker a relationship and bring together people or teams with comple-
During this planning phase grantees can suggest evaluation questions weigh in on terms of reference and help identify potential evaluation consultants
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
29
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Getting the Right EvaluatorEvaluation Team Technical Context Strategy Content and Other Considerations
The Cyber Initiative seeks to cultivate a field that develops thoughtful multidisciplinary solutions to complex cyber challenges and catalyzes better policy outcomes for the benefit of society A couple of years into the strategy the Cyber team commissioned an evaluation42 of its nascent effort to foster a network of cyber policy experts They selected it as an evaluation focus area because of its centrality to the success of the overall strategy and because it offered an early opportunity for learning and course correction
As the team put together a request for proposals they crafted a set of evaluation team criteria Subject matter knowledge of cyber policy was critical for this project as was experience with evaluation and strategy devel-opment so the team invited proposals that would incorporate partnerships between consultants
In the end they selected a proposal in which two firms partneredmdashone with deep evaluation expertise and the other with deep cybersecurity policy knowledgemdashenabling the evaluation to have the degree of rigor an evaluator brings along with cyber content knowledge
If the evaluator doesnrsquot understand the work there can be serious ramifications for building trust accessing relevant subject matter experts recognizing concepts and determining appropriate evaluation designs If the evaluator is exclusively a content expert there can be serious consequencesmdashpredetermined ideas about how things should work a lack of independence and frequently little to no evaluation technical expertise
mentary skills For example when commissioning the evaluations of our Cyber and Nuclear Security ini-tiatives pairing firms that had technical expertise in evaluation with specialists in these respective fields proved valuable Choices always involve trade-offs it is important to manage their risks If we arrange the partnership are we prepared for the extra time it will take for the partners to collaborate Are we and the partners clear on what roles each will play and are the roles well defined and clear
Part of our commitment to diversity equity and inclusion includes consideration for the evalu-ation team with whom we work When selecting an evaluator build in enough time to look for candi-dates from a broad pool of qualified applicants with diverse backgrounds and experiences and reflect on the evaluation teamrsquos ability to draw on knowledge of local context
Grantees can be helpful in the evaluator selection process Including grantees not only may help get them invested in the effort but they also often contribute a useful pragmatic perspective At the very least it is important to introduce a selected evaluator to grantees and let them know of the time com-mitment and expectations for the evaluationmdashand what they can expect in terms of their involvement
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
30
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Selecting an Evaluator
Selecting the right evaluator is hugely important to the success of your evaluation Itrsquos no easy taskmdashan eval-uator is charged with possessing the right mix of evaluation technical expertise content and context knowl-edge and cultural competence The evaluator should understand how to convey evaluation findings to you the client in the ways that work best for you Of course you have a role to play in making that all workmdashbut itrsquos important to select the right partner There are three tips that might help you
bull First think through what qualities are likely to make an evaluation team be successful given your evaluation questions time frame and the context within which the strategy is situated
bull Second talk to more than one evaluation team to understand the variety of approaches they bring to ad-dressing the evaluation questions and collecting and analyzing data Regardless of whether your evaluator is chosen competitively make sure to conduct an interview with them Interviews can help you feel out whether the evaluation team is a good match for your needs
bull And finally one idea is to do a trial run Hire an evaluator to do just part of the evaluation process such as the design phase If both parties feel the partnership is working you can extend the engagement If you donrsquot benefit from working with together you can cut your losses early
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
31
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
ImplementationSometimes an evaluationrsquos implementation does not go precisely as planned Staying connected with the evaluator and the evaluation during implementation can go a long way toward ensuring responsive-ness and a generally higher-quality evaluation
MANAGING AND STAYING INVOLVED WITH AN EVALUATION
As mentioned above less staff time is usually required during implementation of an evaluation while evaluators are collecting data in the field Ongoing management of their work takes some time but in general less than during planning and use That said active management is essential Talk with the evaluator regularly and ask what is or is not going well What are they finding Are the findings making sense Are there data missing or might the data interpretation be off or incomplete due to the complex-ities of the strategy or other issues
Request periodic updates to document progress and any obstacles the evaluator is facing in data collec-tion data quality or other areas These exchanges can be useful forcing functions to keep an evaluation on track and to start troubleshooting early Often the data collection in an evaluation mirrors some of the challenges faced by a program in other facets of its work so evaluation progress updates can be helpful in many ways
Some program staff review and provide feedback on evaluation tools This can be a useful way to ensure that everyone is on the same page about what data will be gathered and why
Support the Evaluatorrsquos Success
As the evaluation commissioner program staff have a significant role in the success of an evaluation whether and how the evaluation ultimately provides information that will be used and shared We hire independent third-party evaluators Therefore it is essential for program staff to
bull Provide the important background information contextual information and introductions to grantees or other key stakeholders to give the evaluators a good chance to gain the requisite knowledge to proceed effectively Help them understand the culture of the foundation and the approach to strategy grantmaking and evaluation For example share these Evaluation Principles and Practices and the Outcome-Focused Philanthropy Guidance
bull Give the evaluator enough time to dig into the background information finalize the evaluation design collect data analyze and interpretmdashand the time and attention to get your feedback It might have taken you a while to plan for the evaluation and to hire the evaluators Allow them the needed time to carry out the evaluation
bull Keep the evaluators apprised of changes to the strategy new information about grantees or issues that develop that may affect either the evaluation design or the interpretation of the findings
Keep in mind Your evaluators should be asking you for information and feedback as they proceed These kinds of conversations ensure that the evaluation is on the right track Donrsquot let too much time go by before you have a conversation about what types of information sharing will be most helpful
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
32
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
It can be especially useful to set an expectation of baseline data summaries or interim evaluation reports on preliminary findings This will keep an evaluation on course engage foundation staff in the discussion of early findings and make space for any needed course corrections For example as part of the evaluation of the Fund for Shared Insight43 the evaluator has produced numerous products ldquoalong the wayrdquo that have been helpful for discussions with funders grantees and prospective funders The evaluations of the Transparency Participation and Accountability and Supporting Local Advocacy in Sub-Saharan Africa strategies similarly have provided materials such as ldquobaselinerdquo summaries and annu-al reports of progress which prompt topics for discussion at convenings
Make sure to take the time to provide updates to the evaluator so they are informed and can adjust In cases where the strategy is evolving and the evaluation is being conducted alongside that evolution it is important for program staff to provide evaluators with timely updates on relevant developments that may affect either the evaluation design or the interpretation of the findings
As important as managing the process is talking through the findings early and often What do they mean Do they resonate Is the evaluator fully aware of the context Is there any reason to make changes to the data collection plan or methodology to capture lessons on emerging areas of interest Here you can consider whether an advisory committee would be worthwhilemdashto bring in others to the discussion of findings and to consider alternative perspectives on how the findings might be used
Using an Advisory Committee to Build Buy-In and Enhance Practicality Quality and Use
Advisory committees are a great way to engage othersmdashfunders grantees other external stakeholders and others internallymdashin an evaluation Developing and using an advisory committee has clear benefits In particular it can help increase the likelihood that the findings are used by and shared more quickly with intended audiences
1 Who You should think critically about who is on the committee and what role they play Which funders grantees and others do you want to have early access to your findings Who can give input on making the evaluation more practical and useful Whose perspectives or voices might be left out
2 Why You can choose your members to serve various purposes You may want to get buy-in from some constituents or feedback on the level of rigor needed to be convincingmdashthis is a way to do it Or sharing internally may be a priority so engaging others internally may help
3 How Remember You determine how and when you want them to engage Typical times are when dis-cussing the design of the evaluation early findings (so they can be your test audiencesounding-board) and recommendations and dissemination Also be mindful of how much you are asking of them consider an honorarium
OTHER CONSIDERATIONS
The advisory committee will need management so it is important to figure out whether this will be part of the evaluatorrsquos responsibility and paid for in the evaluation contract or whether the program officer will be respon-sible If the program officer is responsible be aware that the management takes additional time
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
33
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Continue to check in with and engage grantees in the evaluation A reviewer of this guide said ldquoThe relationship between the evaluators and the implementers is KEYrdquo to successfully conducting an eval-uation and applying the findings in practice If grantees are engaged in the evaluations that touch them they will be (a) more supportive with respect to data collection (b) more likely to learn something that will improve their work (c) less likely to dismiss or defend against the evaluation and (d) better able to help strengthen the evaluation design especially if engaged early From a design perspective this last point is quite important Grantees can serve as a reality check and deepen understanding of the avail-able data and data collection systems They might also suggest potential respondents for interviews or data that may (or may not) be available from their own or othersrsquo monitoring systems As part of the program staffrsquos evaluation management responsibilities it is helpful to check in with grantees about how the evaluation is going for them to get feedback on the process and its strengths and challenges Another idea is to create an evaluation advisory committee that includes representatives from grantee organizations to advise on aspects of the evaluation
RESPONDING TO CHALLENGES
Any number of challenges can emerge during an evaluation A data source may be less reliable than predicted survey response rates may be too low to draw conclusions or consultant staff turnover in the selected firm may reduce confidence in the evaluation team
If you hit these bumps or others in the evaluation road it is important to pause take stock of the chal-lenges revisit prior plans consult appropriate stakeholders consider alternative solutions and make necessary course corrections Donrsquot forget to communicate any changes to everyone invested in the work including grantees
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
34
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use (Including Interpreting and Sharing Findings) Our first evaluation principle is ldquoWe lead with purposerdquo for a reason Establishing a clear purposemdashin-cluding audience use and a timelinemdashsets us up to maximize the usefulness of the evaluations and to live by another of our established principles ldquoWe use the datardquo Not using our findings is a missed op-portunity for learning improvement and course correctionmdashand a waste of time energy and resourc-es And yet using the data requires additional effort including active involvement on the part of the evaluationrsquos intended users and time to reflect and make meaning of the findings We optimize use by sharing the findings using a variety of formats and communication approaches to reach diverse audienc-es Engaging with groups to discuss shared findings taking time to discuss and interpret findings from the evaluation as it progresses and asking yourself ldquoNow whatrdquo go a long way to supporting use
From the very beginning of an evaluation process it is important to plan how the results will be used along the way it is wise to remind yourself of those intended uses
As noted earlier our evaluations tend to have a primary dual purpose of informing Hewlett Foundation staff and grantees and sometimes informing the field Common uses within those categories include informing
bull strategy-level decisions (making course corrections ramping up or strengthening aspects of a strategy testing assumptions or exiting aspects of a strategy)
bull future evaluations (commissioning smaller substrategy or cluster evaluations as inputs to inform a larger strategy-level summative evaluation establishing a baseline or helping to refine targets for change)
bull process improvements (improving data collection grantee proposal or reporting practices and ldquobeyond the grant dollarrdquo activities such as convenings and information sharing)
bull grant and grantee-level decisions (helping to shape renewals of grants closing a grant developing OE grant opportunities switching from project grants to general operating support)
bull other uses (summing up lessons learned as we exit a strategy or substrategy developing frameworks for evaluation and assessment that inform other strategies at the foundation or engaging other funders in discussions of findings)
bull granteesrsquo decisions (for their own program improvement)
bull field use (others learning from what we are doing and how)
Using results is often messier than anticipated Sometimes staff expect more confirmation of suc-cessmdashor for an evaluation to uncover more surprises or ldquoahardquo moments for themmdashthan an evaluation typically delivers Sometimes an evaluation is not especially well done and the results inspire limited confidence Other times staff simply are not sure how to apply the lessons They are uncertain how best to shift or overhaul a strategy or substrategy
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
35
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Use requires that teams pause to reflect and grapple with all of the ldquoWhat So what Now whatrdquo questions Evaluators often provide the ldquowhatrdquo in terms of the findings but the program teams need to grapple with the ldquoso whatrdquo and ldquonow whatrdquo in order to make sure those findings are used This takes dedicated time and effort
Grantees because of their knowledge of content and context play an important role in verifying accuracy and helping interpret and make meaning of the findings Program staff can support use and sharing by convening grantees to discuss findings and sometimes engaging them in a process of co-creating recommendations Or staff can meet with grantees at key junctures when reflection on findings can serve to stimulate ques-tions ideas and opportunities for improvement
Sharing what we learn is essential not only at the conclusion of an eval-uation but throughout the process Sharing is not only a way to ensure that our evaluations maximize their utility it is also consistent with our foundation values and principles of openness and transparency Every program team should plan to publicly share findings from every evalua-tion commissioned in some form
Issues of diversity equity and inclusion are relevant when discuss-ing interpreting and sharing findings Through what lens are the evaluation results interpreted used and shared Who is involved in the discussion If recommendations are made how do these factors come into play Is sharing done in a variety of formats and made accessible to multiple groups
Some Ways to Share (Internally and Externally)
bull Convene grantees to discuss the results and recommendations
bull Organize an internal briefing (eg at a Shoptalk or All Staff) to share with your colleagues what yoursquove learned both about your strategy projects and the evaluation process itself
bull Discuss with your programrsquos board advisory committee how the evaluation results will affirm or inform changes in your strategy or grantmaking approach
bull Share a version of the evaluation with targeted external audiences accompanied by a memo detailing how you are applying the findings in practice
PAUSE TO REFLECT
bull WHAT What did we try with the strategy What results are we seeing
bull SO WHAT What seemed to drive those results (positive and negative)
bull NOW WHAT What do we take away from that How do we apply what wersquove learned going forward
Adapted from Adaptive action Lever-aging uncertainty in our organization44
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
36
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
SHARING RESULTS INTERNALLY
Sharing the results of an evaluation with foundation colleagues brings many benefits and it is worth-while to build this step into your process For staff managing an evaluation these discussions can crys-tallize the results lead to a deeper understanding of those results and force some grappling with what is not yet understood It can also help staff think through what the results mean programmatically and how to apply them in practice For members of other teams review of the evaluation results can gener-ate insights about their own programs grantmaking approaches or evaluation designs
An internal debrief at the end of each evaluation to discuss key lessons learned what went well and what did not and actions taken will help advance the foundationrsquos evaluation practice and keep us fo-cused on designing evaluations with action in mind If another funder has collaboratively supported an evaluation it is often appropriate to consider that partner an internal colleague with respect to sharing results and surfacing implications
Sharing When Findings are Not All Positive
Most times we commission an evaluation we ask evaluative questions to help us and grantees understand both successes and challenges Yet there are times when the findings are more negative than we expected What do we do in those circumstances Consider the following
bull The evaluation officer and communications teams are here to help They can help you plan from the be-ginning what and how to share to address sensitivities and protect reputationsmdashso that we share lessons learned in the most appropriate and effective ways
bull There are external resources that can help you This article45 by BetterEvaluation is a good place to start It includes tips for when you might be in this situationmdashsuch as using a participatory approach from the start limiting surprises discussing the possibility of negative results from the start framing as lessons learned and considering ways to overcome the obstacles that are surfacedmdashbut also emphasizes the need to fully report all findings truthfully even negative ones
SHARING RESULTS EXTERNALLY
Our intention is to share evaluation resultsmdashpositive negative and in-betweenmdashso that others may learn from them Out of respect we communicate with our grantees early on about our inten-tion to share our evaluations and we listen to any concerns they may have about confidentiality Grant agreement letters specify an organizationrsquos likelihood of participating in an evaluation (which as noted above are not typically of just one particular grantee but are usually of numerous grantees who are part of a strategy substrategy or cluster of grants) As an evaluator comes on board it is important to clarify and share with grantees the evaluationrsquos purpose (including any anticipated effect on the grantee) the process for making decisions about it and each partyrsquos required role and participation Itrsquos also import-ant when possible to come to an agreement regarding the level of findings (full evaluation results an ex-ecutive summary or a presentation) that will be shared with which audience You might also negotiate that individualized results will be given to grantee organizations and general results will be shared with a cohort and with selected audiences or publicly
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
37
Evaluation Principles and Practices Practice Guide Planning Implementation and Use
Sharing Evaluation Findings in Ways that Work Well for Different Audiences
The Knowledge for Better Philanthropy strategy supports the creation and dissemination of knowledge about how to do philanthropy well From an earlier evaluation the program team learned that the grant-ees were producing high-quality knowledge and distributing it widely But they were missing key piec-es of information how foundations find knowledge resources and whether these knowledge products inform or influence their philanthropic practice These pieces are central to the strategy but had never been systematically assessed As the philanthropy grantmaking team embarked on this new evaluation they took time to ask the grantees what they would like to learn as part of the evaluation included their questions where possible involved them in an evaluation advisory committee and established a process for sharing the findings
The evaluators produced a summary report46 highlighting key findings But the team did not stop there First the program officer hired a graphic design firm to help translate the report findings into easily digest-ible bites47 This was in fact a finding from the study itself Funders prefer easily digestible formats that are practically applicable and relevant to their work These resulted in easily shareable visually friendly snapshots of the data Second the program officer ensured that the grantees who were included in the study were also able to make best use of the work To do so each grantee received a confidential indi-vidualized report which included that organizationrsquos data as compared to othersrsquo (presented anonymous-ly) These two extra stepsmdashmaking the work more visually accessible and relevant reporting to grantees who participatedmdashled a grantee to publish this commendation48 Finally a Foundation Commissioned Study Which Actually Helps its Grantees
Doing this was not free of cost To prepare both the public and the individualized reports cost more time and more money It also required both upfront planning and some level of flexibility The team didnrsquot know exactly how they hoped to share the findings right at the start but they built in the financial and timeline cushion to explore They ultimately had to amend their contract with the evaluators to make this possi-blemdashbut to the grantees involved in the Knowledge work and the broader field these steps made the report more useful and relevant
The program officer also looped in the Hewlett Foundation communications officer who was able to provide input in a timely way about effective email web and social sharing
We consider the question of in what form we will be share evaluation findings on a case-by-case basis with care given to issues of organizational confidentiality For instance if an evaluation is in part focused on questions of organizational development it may be more useful for the findings to be shared in full with that grantee so the grantee can use the results to drive improvement without having to take a defensive public stance and also to work with the evaluator to surface broader lessons to be shared publicly
The foundation expects that some productmdashwhether itrsquos a full evaluation report an executive summary or a presentationmdashfrom the process will be made public to support openness trans-parency and learning When planning how to share results publicly program staff should consult with the foundationrsquos communications staffmdashideally early in the process and over the course of the evalua-tionmdashto determine the best approach They should review documents for sensitive issues and flag those for legal review before sharing results publicly
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
38
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
Key Terms
ACTIVITIES The actions taken by the foundation or a grantee to achieve intermediate outcomes and make progress toward the achievement of goals [Gates Foundation glossary]
BASELINE An analysis or description of the situation prior to an interven-tion against which progress can be assessed or comparisons made [Gates Foundation glossary]
EVALUATION An independent systematic investigation into how why to what extent and for whom outcomes or goals are achieved It can help the foundation answer key questions about strategy substrategy clusters of grants or sometimes a single grant [Variant of Gates Foundation glossary]
DEVELOPMENTAL A ldquolearn-by-doingrdquo evaluative process that has the purpose EVALUATION of helping develop an innovation intervention or program
The evaluator typically becomes part of the design team fully participating in decisions and facilitating discussion through the use of evaluative questions and data [Variant of The En-cyclopedia of Evaluation (Mathison 2005) and Developmental Evaluation (Quinn Patton 2011)]
FORMATIVE An evaluation that occurs during a grant initiative or strategy EVALUATION to assess how things are working while plans are still
being developed and implementation is ongoing [Gates Foundation glossary]
IMPACT A type of evaluation design that assesses the changes that EVALUATION can be attributed to a particular intervention It is based on
models of cause and effect and requires a credible counter-factual (sometimes referred to as a control group or compar-ison group) to control for factors other than the intervention that might account for the observed change [Gates Founda-tion glossary USAID Evaluation Policy]
PERFORMANCE A type of evaluation design that focuses on descriptive or EVALUATION normative questions It often incorporates beforeafter com-
parisons and generally lacks a rigorously defined counterfac-tual [USAID Evaluation Policy]
SUMMATIVE An evaluation that occurs after a grant or intervention is EVALUATION complete or in service of summing up lessons to inform a
strategy refresh or when exiting a strategy in order to fully assess overall achievements and shortcomings [Variant of Gates Foundation glossary]
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
39
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
EVIDENCE A general term that refers to qualitative and quantitative data that can inform a decision
FEEDBACK Data about the experience of the people who we hope will ultimately be positively touched by our work Feedback can provide new information and insight that can be a valuable source for learning while it may inform evaluation it is a dis-tinct channel
GOAL A general statement of what we want to achieve our aspira-tion for the work [OFP Guidebook]
OUTCOME Specific change we hope to see in furtherance of the goal
IMPLEMENTATION A catch-all term referring to particular activities MARKER developments or events (internal or external) that are useful
measures of progress toward our outcomes and goal
GRANT A sum of money used to fund a specific project program or organization as specified by the terms of the grant award
INDICATORS Quantitative or qualitative variables that specify results for a particular strategy component initiative subcomponent cluster or grantee [Gates Foundation glossary]
INITIATIVE A time-bound area of work at the foundation with a discrete strategy and goals Initiatives reside within a program despite occasionally having goals distinct from it (eg the Drought Initiative within the Environment Program)
INPUTS The resources used to implement activities [Gates Foundation glossary]
LOGIC MODEL A visual graphic that shows the sequence of activities and outcomes that lead to goal achievement [OFP Overview]
METRICS Measurements that help track progress
MONITORING A process that helps keep track of and describe progress to-ward some change we want to seemdashour goals or outcomes Evaluation will often draw on monitoring data but will typically include other methods and data sources to answer more strategic questions
MampE An acronym used as shorthand to broadly denote monitoring and evaluation activities It includes both the ongoing use of data for accountability and learning throughout the life of a grant component initiative or strategy as well as an examination of whether outcomes and impacts have been achieved [Gates Foundation glossary]
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
40
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
OUTCOME-FOCUSED A framework that guides how we do our philanthropic work PHILANTHROPY from start to finish It reflects the foundationrsquos commitments (OFP) to being rigorous flexible adaptive transparent and open
while staying focused on results and actively learning at every juncture [OFP Overview]
STRATEGY A plan of action designed to achieve a particular goal
SUBSTRATEGY The different areas of work in which a program decides to invest its resources in order to achieve its goals Each substrategy typically has its own theory of change implemen-tation markers and outcomesmdashall of which are designed to advance the programrsquos overall goals
CLUSTER A small group of grants with complementary activities and objectives that collectively advance a strategy toward its goal
TARGETS The desired level for goals the program plans to achieve with its funding They are based on metrics and should be ambi-tious but achievable within the specified time frame
THEORY OF A set of assumptions that describe the known and CHANGE hypothesized social and natural science underlying the graph-
ic depiction in a logic model [OFP Overview]
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
41
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX A Evaluate Throughout a Strategy
CONTINUE EVALUATING
EVALUATE
EVALUATE
EVALUATE
EVALUATE
ORIGINATE
EXIT
IMP
LEM
ENT
REFR
ESH
Develop an evaluation plan
bull What are your most important evaluation questions for the new strategy
bull What are the key assumptions of the new strategy
bull What areas of the strategy (substrategy clusters of grants) are important to evaluate When will findings be most valuable and why
bull Commission strategy substrategy and cluster evaluations of key areas of the overall strategy
bull These may be developmental formative or summative based on the questions and timing for decision-making
bull After refresh develop a new evaluation plan and prioritize and sequence evaluations
bull Synthesize findings across evaluations commissioned to date
bull Commission evaluation to fill in the gaps if needed
bull If exit commission an evaluation to sum up accomplishments and key lessons learned
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
42
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX B Editorial Guidance What Evaluators Need to Know
Consistent with the value the Hewlett Foundation places on openness and transparency we are com-mitted to sharing the results of evaluations of our grantmaking so that others may learn from our experience and hold us accountable for the results of our work In fact we presume that results from all evaluations will be shared publicly via our website with only limited principled exceptions where sharing could cause material harm to the foundationrsquos grantees or our strategies
In order to facilitate the foundation review and publication of the evaluation you are preparing for us we ask you to keep the following points in mind as you draft your report and any related products They reflect the most common requests we make for edits to draft evaluation reports and we hope that mak-ing you aware of them in advance will help streamline the editing and publication process
Provide accurate descriptions of grantee advocacy efforts As you may know as a private foundation the Hewlett Foundation must comply with legal rules that preclude using our grant funds for lobbying and partisan election activities Thatrsquos the short description The actual rules regarding grantee lobbying and election activities are quite complicated and it is important that evaluations of our work reflect what is and is not permissible in our grant agreements We ask that you flag for us in your draft report any sections where you discuss lobbying or election activities and also note (either in the body of the report or a footnote) that while the report may describe grantees efforts to affect legislation or govern-ment policy the Hewlett Foundation does not earmark its funds for prohibited lobbying activities as defined in federal tax laws and that the foundation does not fund partisan electoral activities though it may fund nonpartisan election-related activities by grantees
Provide accurate descriptions of fundergrantee relationship The Hewlett Foundation believes strongly in treating our grantees as partners rather than contractors carrying out our directives They are the ones with the expertise and front-line perspective and we strive to work with them in ways ldquothat are facilitative rather than controllingrdquo as our guiding principles49 put it Because evaluations we commission often look through the lens of our grantmaking strategy the nature of our relationship with grantees is sometimes lost and grantees are portrayed in a manner that is more instrumental than col-laborative Itrsquos accurate to describe shared goals between the foundation and our grantees but we ask that you avoid descriptions that paint us as ldquopuppet mastersrdquo or strategists moving pieces on a chess board To be sure we may not always achieve our goals regarding collaboration and partnership and if itrsquos accurate and appropriate to report that please do so However we do not describe our granteesrsquo work or achievements as our own and we prefer that evaluations not do so either It is the work and achievements of grantees that we have strategically chosen to support
Avoid undue flattery Therersquos a natural tendency in preparing a report for a client to sing the praises of that client Sometimes that results in puffery evaluators giving undue praise or trying to flatter the foundation This is wholly unnecessary and a bit off-putting It also conflicts with our goal in commis-sioning an evaluation which is to get constructive independent feedback on work we support so that we and others can learn and improve We ask that you strive for a dispassionate and measured tone acknowledging with candor what we got right and what we got wrong
Criticism of or confidential information about individual grantees Two exceptions to our general rule about openness and transparency are information that we have an ethical or legal duty to keep confidential (eg staffing changes at a grantee that have not yet been made public) or situations where sharing information publicly could cause material harm to a grantee such as criticism of an individual
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
43
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
organizationrsquos work For that reason we ask that you include whatever such information is relevant to your report but flag it for us in a draft version so we can consider how best to fulfill our ethical and legal obligations to our grantee partners as we share the results in targeted fashion with other partners or more broadly with the field and the public
Thank you in advance for taking these points under advisement as you draft your evaluation reports and related products
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
44
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX C Engage Grantees in EvaluationP
LAN
NIN
G
Get input from grantees about what they hope to learn from an evaluation
Build grantee questions into the evaluation when possible
Share draft RFP or Evaluation Purpose and solicit feedback
Be clear about how the results of the evaluation will be used and shared publicly
If appropriate provide opportunity to weigh in on evaluator selection process
Consider whether there will be an advisory group for the evaluation and how grantee representatives might be involved
IMP
LEM
ENTI
NG
Introduce the external evaluator before the evaluation begins
Be upfront from the start about the demands of the evaluation (ie share a FAQ)
Ask for input on methodology and timing of data collection activities
Keep in touch with grantees about upcoming evaluation activities
Check in about the evaluation process along the way how is it going for them
Share and discuss relevant findings
US
ING
+ S
HA
RIN
G
Share findings with grantees and get feedback on findings and evaluatorrsquos interpretation
Consider opportunities to co-create relevant recommendations
Provide grantees with the opportunity to verify accuracy of data before finalizing
Discuss how findings might be used
Brainstorm settings and opportunities to share findings together
Convene grantees to discuss the results and recommendations
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
45
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
APPENDIX D 7 Principles of Evaluation
1
2
3
4
5
6
7
We lead with purpose
We use the data
Evaluation is fundamentally a learning process
We cannot evaluate everything so we choose strategically
We treat evaluations as a key part of the strategy lifecycle
We maximize rigor without compromising relevance
We share what we are learning with appropriate audiences and share publicly
By anticipating our information needs we are more likely to design and commission evaluations that will be useful and used
bull Design evaluation with actions and decisions in mind
bull Ask how and when will we and others use the information that comes from this evaluation
Establishing evaluation questions early in the strategy lifecycle helps us clarify and refine how why for whom when and to what extent objectives or goals are expected to be achieved
bull Actively learn and adapt as we plan implement and use evaluations
bull Use evaluation as a key vehicle for learning as we implement our strategies to bring new insights to our work
Undergoing an evaluation either of the whole strategy or of a key part within three years ensures we donrsquot go too long without external eyes
bull Criteria guide decisions about where to put our evaluation dollars includ-ing opportunity for learning urgency to make course corrections or future funding decisions the potential for strategic or reputational risk and size of investment as a proxy for importance
Selecting evaluation designs that use multiple methods and data sources when possible strengthens our evaluation designs and reduces bias
bull Match methods to questions and do not routinely choose one approach or privilege one method over others
bull Evaluations clearly articulate methods used and their limitations
bull Evaluations include comparative reference points
We presumptively share the results of our evaluations so that others may learn from our successes and failures
bull Identify audiences for findings as we plan
bull Communicate early with our grantees and co-funders about intention to evaluate and plans to share
bull Share the results of our evaluations in some form (full executive summary or presentation) with care given to issues of confidentiality
Not using findings is a missed opportunity for learning and course correction
bull Take time to reflect on the results generate implications for our strategies grantees policy or practice and adapt
bull Combine the insights from evaluation results with wisdom from our own experiences
Building evaluative thinking in throughout the strategy lifecycle helps artic-ulate key assumptions in a theory of change or strategy and establishes a starting point for evaluation questions and a proposal for answering them in a practical meaningful sequence
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
46
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
1 Evaluation Principles and Practice First Edition httpswwwhewlettorglibraryevaluation-princi-ples-and-practices
2 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
3 Hewlett Foundation Guiding Principles reference to Outcome-Focused Approach httpshewlettorgout-come-focused-approach
4 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
5 Tracking Progress Setting Collecting and Reflect-ing on Implementation Markers July 2018 httpshewlettorgwp-contentuploads201807OFP-Guide-Tracking-Progresspdf
6 ldquoTime for a Three-Legged Measurement Stool Going beyond traditional monitoring and evaluation to focus on feedback can lead to new innovations in the social sectorrdquo Fay Twersky SSIR Winter 2019 httpsssirorgarticlesentrytime_for_a_three_legged_measure-ment_stool
7 Outcome-Focused Philanthropy httpwwwhewlettorgwp-contentuploads201612OFP-Guidebookpdf
8 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
9 Hewlett Foundation Guiding Principles reference to Diversity Equity and Inclusion Principle hewlettorgdiversity-equity-inclusion
10 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
11 Hewlett Foundation Guiding Principles reference to Openness Transparency and Learning httpshewl-ettorgopenness-transparency-learning
12 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
13 ldquo5-A-Day Learning by Force of Habitrdquo httpsme-diumcomjcoffman5-a-day-learning-by-force-of-habit-6c890260acbf
14 The Value of Evaluations Assessing spending and quality httpshewlettorgvalue-evaluations-assess-ing-spending-quality
15 American Evaluation Association Guiding Principles For Evaluators httpwwwevalorgpcmldfid=51
16 Evaluation of The William and Flora Hewlett Foundationrsquos Organizational Effectiveness Program Final Report November 21 2015 httpshewlettorgwp-contentuploads201610Evaluation-of-OE-Pro-gram-November-2015pdf
17 ldquoAssessing nonprofit capacity A guide to toolsrdquo Prithi Trivedi and Jennifer Wei October 30 2017 httpshewlettorgassessing-nonprofit-capaci-ty-guide-tools
18 ldquoEvaluating the Madison Initiativerdquo Daniel Stid January 24 2019 httpshewlettorglibraryevaluat-ing-the-madison-initiative
19 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
20 Evaluation Final Report Hewlett Foundationrsquos strategy to apply human-centered design to family planning and reproductive health Itad July 24 2018 httpshewlettorglibraryevaluation-of-the-hew-lett-foundations-strategy-to-apply-human-cen-tered-design-to-improve-family-planning-and-re-productive-health-services-in-sub-saharan-africa
21 ldquoPromise and progress on family planning in Fran-cophone West Africardquo Margot Fahnestock July 25 2018 httpshewlettorgpromise-and-prog-ress-on-family-planning-in-francophone-west-africa
22 International Womenrsquos Reproductive Health Support-ing Local Advocacy in Sub-Saharan Africa April 2016 httpshewlettorgwp-contentuploads201611Sup-porting-Local-Advocacy-in-Sub-Saharan-Africapdf
23 Western Conservation Strategy 2018-2023 July 16 2018 httpshewlettorglibrarywestern-conserva-tion-strategy-2018-2023
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
47
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
24 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
25 Research on Open OER Research Hub Review and Futures for Research on OER Linda Shear Barbara Means Patrik Lundh October 14 2015 httpshew-lettorglibraryresearch-on-open-oer-research-hub-review-and-futures-for-research-on-oer
26 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
27 The Hewlett Foundation Education Program Deeper Learning Review Executive Summary January 30 2014 httpshewlettorglibrarythe-hewlett-founda-tion-education-program-deeper-learning-review-ex-ecutive-summary
28 Deeper learning six years later Barbara Chow April 26 2017 httpshewlettorgdeeper-learning-six-years-later
29 The William and Flora Hewlett Foundationrsquos Nu-clear Security InitiativemdashFindings from a Summa-tive Evaluation ORS Impact May 4 2015 httpshewlettorglibrarythe-william-and-flora-hewl-ett-foundations-nuclear-security-initiative-find-ings-from-a-summative-evaluation
30 ldquoThe Legacy of a Philanthropic Exit Lessons From the Evaluation of the Hewlett Foundationrsquos Nuclear Security Initiativerdquo Anne Gienapp Jane Reisman David Shorr and Amy Arbreton The Foundation Review Vol 9 Iss 1 Article 4 2017 httpsscholar-worksgvsuedutfrvol9iss14
31 ldquoQampA with Margot Fahnestock A teen-centered ap-proach to contraception in Zambia and Kenyardquo Sarah Jane Staats July 25 2018 httpshewlettorgqa-with-margot-fahnestock-a-teen-centered-approach-to-contraception-in-zambia-and-kenya
32 ldquoBetterEvaluation communityrsquos views on the difference between evaluation and researchrdquo December 2014 Patricia Rogers httpswwwbetterevaluationorgenblogdifference_between_evaluation_and_research
33 Improving the Practice of Philanthropy An Eval-uation of the Hewlett Foundationrsquos Knowledge Creation and Dissemination Strategy Harder + Co November 2013 httpshewlettorglibraryan-eval-uation-of-the-knowledge-creation-and-dissemina-tion-strategy
34 Evaluation of the Population and Poverty Research Initiative (PopPov)Julie DaVanzo Sebastian Linne-mayr Peter Glick Eric Apaydin 2014 httpshewlettorgwp-contentuploads201608RR527_REVFINAL-COMPILEDpdf
35 Best Practices for Enduring Conservation Hov-land Consulting July 2018 httpshewlettorglibrarybest-practices-for-enduring-conserva-tion-five-year-retrospective-of-the-hewlett-founda-tions-western-conservation-grantmaking-strategy
36 A new conservation grantmaking strategy for todayrsquos challenges Andrea Keller Helsel July 16 2018 httpshewlettorga-new-conservation-grantmak-ing-strategy-for-todays-challenges
37 Contribution Analysis httpswwwbetterevaluationorgenplanapproachcontribution_analysis
38 Process Tracing httpswwwbetterevaluationorgevaluation-optionsprocesstracing
39 Qualitative Comparative Analysis httpswwwbetterevaluationorgevaluation-optionsqualitative_comparative_analysis
40 Quip httpswwwbetterevaluationorgenplanap-proachQUIP
41 Taking stock of our Performing Arts grantmaking John McGuirk April 2016 httpshewlettorgtaking-stock-of-our-performing-arts-grantmaking
42 Evaluation of Network Building Camber Collective November 2016 httpshewlettorgwp-contentuploads201802Evaluation-of-network-building-Cy-ber-2016pdf
43 Evaluation findings httpswwwfundforsharedin-sightorgknowledget=evaluating-our-workknowl-edge-tabs|3
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies
48
Evaluation Principles and Practices A Guide to Evaluation at the William and Flora Hewlett Foundation
44 Adapted from Adaptive action Leveraging uncertain-ty in our organization G Eoyang R Holladay 2013 Stanford University Press
45 52 weeks of BetterEvaluation Week 23 Tips for de-livering negative results Jessica Sinclair Taylor June 2013 httpwwwbetterevaluationorgblogdeliver-ing-bad-news
46 Peer to Peer At the Heart of Influencing More Effec-tive Philanthropy A Field Scan of How Foundations Access and Use Knowledge Harder+Co and Edge Research February 2017 httpwwwhewlettorgwp-contentuploads201703Hewlett-Field-Scan-Re-port-2017-CCBYNCpdf
47 Peer to peer At the heart of influencing more effec-tive philanthropy February 2017 httpshewlettorgpeer-to-peer-at-the-heart-of-influencing-more-effec-tive-philanthropy
48 Finally a foundation-commissioned study that actual-ly helps its grantees by Aaron Dorfman March 2017 httpswwwncrporg201703finally-foundation-com-missioned-study-actually-helps-granteeshtml
49 Hewlett Foundation Guiding Principles httpswwwhewlettorgabout-usvalues-and-policies