Monitoring and Evaluation of Influence Enrique Mendizabal June 2011.

Monitoring and Evaluation of

Influence

Click icon to add partner logo Click icon to add partner logo Click icon to add partner logo Click icon to add partner logo

Enrique Mendizabal

June 2011

Outline

• The skeptical view

• Why monitor and evaluate?

• What can we monitor and evaluate?

• Types of influencing

• Methods and tools for each

• A suggestion

• Work on your own

The skeptics

3

(on think tanks) some experts say:

From Braml (2004)

• Kent Weaver: it is hard to determine policy influence of one think tank in relation to others because they are all different, work on different aspects of a policy, influence in different ways and there are many other players involved.

 • Nelson Poslky: there is no sense in looking for direct

influence of think tanks’ activities because one can only ask these questions when one ignores the complexity of political processes. Some causal relationships may only be found in a few cases, but systematic explanations of this sort remain an illusion.

5

more

Weidenbaum (2009)

• Andrew Rich: dollar for dollar, think tanks attract much more attention than any other organisation.

 • John Hamre, of the Center for Strategic and International Studies:

evaluating the influence of think tanks on a particular policy would be like determining who is the father of Anna Nicole Smith’s baby. As soon as the change happens every think tank who had something to say about it is quick to claim it for itself.

 • David Frum, ofthe American Enterprise Institute: since one cannot

measure influence, [no method] would do it.

6

even more

Abelson (2006)

• Abelson: by focusing on the influence of think tanks on policy change one would overlook a great deal of other types of influence that is the source of think tanks’ value.

 • Leslie Gelb, form the New York Times: influence is highly

episodic, arbitrary and therefore difficult to predict.  • Abelson and Ricci: as the policy community becomes more

open and complex it also becomes more difficult to say anything meaningful about the contribution of think tanks to it.

7

and more

Puryear (1994) • The most interesting and valuable contributions of

think tanks [in Chile] may not be intellectual but psychological: the hundreds of events and seminars that Chilean think tanks organised throughout the 1980s helped to restore the mutual trust and understanding missing from Chilean politics and that had led to the rupture of democratic order.

8

And:

Hoppe (2010)

The search for ‘impact’ is driven by three assumptions:• Unidirectional transfer of advice;• Policymakers and experts performing different

and exclusive tasks;• All possible impact is desirable per se

9

In practice, however:• Robust research into uptake and impact will be so

costly and time consuming, that nobody would be able to afford it.

• Lessons drawn will be marginal and irrelevant for other situations.

• Indicators focusing on that can be measured would only lead to conformance and perverse conduct.

Why monitor and evaluate?

10

Why should you do M&E?

• To learn• To manage better• To get more funds• To keep funders/clients happy

• (“what is the guarantee that doing this actually helps us?”)

Researchers vs research

Do we want:

1) Our research/IE to be more influential?

or

2) Policy to be based on evidence?

Visibility or substance?Visibility Substance

Short term ‘relevant’ research Long term research

Focus on solutions for ‘agreed’ problems

Engage with the definition of the problem

Media exposure Lobby, network, horse trading

Briefing papers, Opinion pieces Estimates, costed proposals, policy options

Website, Blogs, Facebook, etc. Academic publications, long reports

Online communities with millions of hits

Communities with the right people

Delegations at high level global conferences

Private meetings at Party conferences and private meetings while planning for the high level conferences

Event focused influence Problem focused influence

Global Go-To-Survey Prospect magazine Think Tank of the year

What are we evaluating? –and

some complications

14

1. Strategy and

M&E & Learning

1. Strategy and direction –are you doing the right things?

2. Management –are you doing what you planned to do?

3. Outputs – are outputs up to scratch?

4. Uptake – are people aware of your work?

5. Outcomes and impacts –are you having any impact?

16

How?

Hovland (2009):

1. Strategy and direction: Logframes; Social Network Analysis; Impact Pathways; Modular Matrices

2. Management: ‘Fit for Purpose’ Reviews; ‘Lighter Touch’ Quality Audits; Horizontal Evaluation; Appreciative Inquiry

3. Outputs: Evaluating academic articles and research reports; Evaluating policy and briefing papers; Evaluating websites; Evaluating networks; After Action Reviews

4. Uptake: Impact Logs; New Areas for Citation Analysis; User Surveys

5. Outcomes and impacts: Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies

M&E of policy influence – what’s the problem?

POLICY

Influencing

Activities

And who does the influencing?

• The researchers• Campaigners/third parties?• Policymakers themselves?

Losing control

Inputs Activities Outputs

Other Actors

Project Team

Outcomes

ImpactOutcomes ImpactOutcomes Impact

Policy changes

24

So we need to look for proxys

“Government runs in part, on the basis of memos. If a SD or DD official, or an analyst at the CIEA or the NSC, has your study in front of him and open at the time he is writing his own memo to the secretary or the director or perhaps the president himself –if, in short, he is using your ideas and analysis at the time he writes his own memo- then you have influence. If your study is not open in front of him, or worse, you do not even know who the responsible official is, you do not have influence. It is as simple as that.”

Howard Wiarda quoted by Abelson D. E., 2006

Types of influencing

What to measure and how

3 types of influencing approaches (there are more):

• Evidence and advice• Public debate/education and

advocacy• Lobbying approaches

Methods and tools

Evidence and advice

How? What to measure Tools

Consultancy

Research and analysis, ‘good

practice’

Evidence-based argument

Providing advisory support

Developing and piloting new policy

approaches

OutputsEvaluating research reports, policy briefs

and websites

Uptake and use

Logs;

new areas for citation analysis;

user surveys

Influence

RAPID outcome assessment;

Episode studies;

Most Significant Change

Public debate/education and advocacy

How? What to measure Tools

Public communications and

campaigns

University teaching

‘Public education’

Debate

Messaging

Advocacy

Target audience attitudes, behaviour,

etc

Surveys,

focus groups,

direct responses, career paths

Media attentionMedia tracking logs,

media assessment

Media framing and influence

Framing analysis; coverage

Lobbying approaches

How? What to measure Tools

Face-to-face meetings and discussions

Relationships and trust

Direct incentives and diplomacy

Actors;

relationships;

policy process and institutions

Recording meetings;

tracking people;

interviewing key informants;

probing influence

Recommendations

• Develop a Theory of Change based on sound theory

• Integrate PME for policy influence

• Use a mix of tools to gather data

• Ask: What does it mean?

A suggestion

33

An example (based on ODI’s structure)

Develop your own

Exercise

• Describe the change you would like to see • Describe the role you will play in it -your contribution

to this change • How will you know that you have fulfilled this role to

the best of your abilities?• How will you know whether you are doing the right

thing? • How will you incorporate lessons (that you and others

have learned) into your planning and implementation?

Additional resources

www.odi.org.uk/rapidwww.ebpdn.org

www.outcomemapping.ca

www.onthinktanks.org