Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times

Impact Assessment E2B

Govert Gijsbers -TNO

2

Overview Workshop

Programme

• 10.00-10.45 Introductory presentation

• 10.45-11.15 Discussion

• 11.15-11.30 Break

• 11.30-12.30 Towards indicators: Generating

ideas and indicators on long-term impact,

outcomes and outputs

• 12.30-12.45 Wrap-up

3

Introductory Presentation

• An impact case

• Some methodological issues

• The impact assessment process

I. Mini case study

Dr. Pasveer’s Ditch

5

The “Pasveer sloot”

1950s: Sewage treatment was expensive and only available for larger

cities, no cost-effective solution for smaller towns existed.

Dr. Pasveer at TNO set out to develop a low cost solution, mimicking

natural processes in rivers, in a closed system:

• An „oval‟ ditch with an inlet for sewage and an outlet for treated water

• A rotor and brushes to circulate the water and to add oxygen

• Storage and treatment for sludge

• First pilot at TNO in Delft

6

1954: The cutting edge of environmental technology!

- Fully operational plant (capacity 600 user equivalents)

- Excellent technical performance: much better than existing systems

- Cheap: 50% lower cost than existing technology

7

Scaling up and scaling out

- 400,000 user equivalents!

- World-wide adoption

8

1990s: Disadoption -Now an industrial monument

- Stricter norms for emissions (fully aerobic process, energy-

intensive, CO2 emissions)

- Huge surface area required for large scale treatment facilities

- But not the end of the story…

9

Technical improvements: Carousel

model

• Underlying technology is the same

• Used globally today

10

Huge impact – with hindsight (1)

• Initially hard to recognize as a breakthrough technology

• Long term perspective: 1954-1991 (NL)

– still in use in other countries, and the basis for current

technology

• TNO didn‟t commercialize it – never saw a penny in

return

• Dutch engineering companies built systems all over the

world and profited handsomely

• Is that a problem? Public money, public goods?

11

Huge impact – with hindsight (2)

• Huge benefits; different types: financial, environmental,

public health

• How to measure?

– Worldwide spillovers

– Impact on subsequent technology (carousel model)

• Finally: Who cares?

– Then was then and now is now!

– Past performance not a criterion for future funding

– Key issue: IA can only be done after the fact, but then

stakeholders (policy makers) have lost interest.

12

A few lessons

• Impact may take a long time to emerge

– but it may also persist for a long time

• Different types benefits/impacts

– financial, environmental, public health

• Technology adoption can be world-wide

• More actors involved: R&D, companies:

– attribution

• Subsequent technical improvements

II. Some methodological

issues

14

Why Impact Assessment?

• R&D / innovation programmes challenged to

demonstrate their impact on clients, stakeholders and

society

• Two main reasons for impact assessment (IA)

– Accountability (external)

– Learning (internal)

• Much talk about IA, but not much practical work

– Because IA is a difficult issue

• Challenge to develop a workable system

15

Eight issues in IA

1. Level of analysis– Impact of what?

2. Level of analysis– Impact on what?

3. Attribution

4. Time lags

5. Selection bias

6. Non-linearity

7. The problem of the counterfactual

8. Measurement issues

16

1. Level of analysis – Impact of

what?

– Projects, Programs, Technologies, Business Units,

Departments, Organisations?

– Is the impact of the whole greater than the sum of the

parts?

17

2. Level of analysis – Impact on

what?

– Direct outcomes for clients

– Indirect outcomes for clients and stakeholders

(intended and unintended)

– Impact on society at large

18

3. Attribution• Attribution (of impact) a key problem in IA

• With every step away from the direct outputs produced by

an organization or program, the question of attribution

becomes more pertinent

19

The Attribution Gap

20

“Contribution analysis”(to deal with attribution problems)

“Contribution analysis focuses explicitly on the issues of

attribution and contribution and aims to present a

credible story of how a program or project contributes

to solving a problem.” (John Mayne, Canadian Auditor

General Office )

• “Plausible association”: whether “a reasonable person,

knowing what has occurred in the program and that the

intended outcomes actually occurred, agrees that the

program contributed to those outcomes?”

21

4. Time lags

• New technologies may take decades to impact on

society

• Looking back 25 years at past achievements is no longer

of much policy interest

22

5. Selection bias

• “Unsuccessful” projects not well documented

• Needs an estimate of the distribution of project success

rates (representative portfolio)

• Alternatively, focus on a few “blockbusters”

– R&D inherently an uncertain business, where many

project must fail

– a few mega successes may repay the cost of R&D

many times over

23

6. Non-linearity

• The impact chain presents a linear model

– a bit like the linear model of innovation

• But we know that impact is often achieved in roundabout

and unexpected ways

24

7. The problem of the

counterfactual

• We don‟t know what would have happened

anyway (without the project / program in

place)

• With / without

• Control groups?

• (Behavioural) additionality

25

8. Measurement issues

• Quantitative and qualitative approaches– from econometric models to story telling

• Quantification limited by conceptual and data problems

• R&D produces “knowledge”, which is hard to measure, but is more than the number of patents.

• Need to develop indicators that can be measured

• Start with Identification of all outputs (not a trivial task for a major programme such as E2B)

• Numbers supported by qualitative stories

III. Developing an IA process

27

Key steps

1. Define IA requirements

• with key stakeholders

2. Build an E2B “logic model”

– inputs, outputs, outcomes, impact

3. Measurement:

– ID indicators for outcomes

4. Implementation

• responsibilities, etc.

28

1. Impact assessment system

requirements E2B

• Provides relevant information on E2B‟s performance (outputs, outcomes and impact) to different users

• Is accepted by E2B funding agencies as a credible set of instruments

– this is the political aspect

• Is cost-effective in information collection and maintenance

• Strong links to E2B mission and strategic objectives

• Builds on best practice from public and private sector, nationally and internationally.

29

2. Build a “logic model”

• A logic model connects the inputs and outputs of

a project or programme to intermediate and final

outcomes (impact)

• A.k.a.

– program logic

– intervention logic

– outcomes model

– theory of action

– impact pathway

– mental model of the program

30

InputsActivities/Process

Intermediate

Outcomes

Products /Outputs

• Finance

• People

• Infrastructure

• R&D and

innovation

projects

• Technologies

• Publications

• Business

models

• Prototypes

• Services

• Technologies

/ materials

adopted

• Houses

insulated /

energy neutral

• Construction

processes

adopted

• Reduced GHG

emission

Program logic - general

Final

outcomes /Impact

• Incomes

• Employment

• Environment

• Climate

change

• Public health

• Cultural

heritage

31

Example: Overview model

32

Sub model - training

33

Logic model

34

3. Develop indicators

• Start with outputs

– basically counting, but…

– many R&D organisations/ programs do record all

outputs

• Add indicators to outcomes

– (to be) collected by program, or

– collected by others (statistics offices)

• ID “contributions” to aggregate societal/economic impact

• Key point: indicator development comes at the end, not

the beginning

35

4. Implementation

• Assign responsibilities to project partners

• Set up (annual) monitoring, data collection

– primary, secondary

• Complement quantitative indicator-based

information with in-depth case studies on

specific topics

• Communication of impact to funders,

clients, stakeholders

36

Overview Workshop

Programme

• 10.00-10.45 Introductory presentation

• 10.45-11.15 Discussion

• 11.15-11.30 Break

• 11.30-12.30 Towards indicators: Generating

ideas and indicators on long-term impact,

outcomes and outputs

• 12.30-12.45 Wrap-up

E2B expected impacts

38

Impacts: Energy

• New zero energy, energy+ buildings

• Renovated energy efficient buildings

• Increased RES installments

• Reduced energy dependence

– reduced energy consumption

39

Outcomes: environmental

• Reduce CO2 emissions

– 65 Mt /yr

• Low energy content materials

• Reduction of waste, 75% recycling

• Municipal solid wast reduction

– 26 Mtoe/yr

40

Outcomes: social

• 90-150k jobs

• Improved indoor climate:

– CO2 content 400 ppm, humidity 30-50%

• Sound climate (noise)

• Lighting comfort & reduced energy

consumption (using ICT)

• Accessibility

41

Outcomes: Economic

• Investment in efficient & clean technologie

of 12-17,000 M €/yr

• Cost reduction of 10-15% on above

investment

• Shorten time to market

• Increased labor productivity

• Cost savings: lighting, heating, water

– 126-150,000 M€

42

Outputs, outcomes and impact

43

“Contribution analysis”

Contribution Analysis: Addressing Attribution with

Performance Measures

• Acknowledge the attribution problem

• Present the logic of the program – “theory of

action”, “impact pathways”

• Identify and document behavioral changes (clients

or beneficiaries)

• Use discriminating indicators.

• Track performance over time.

• Discuss, and test alternative explanations.

• Gather additional relevant evidence.

• Gather multiple lines of evidence.