Economic Evaluation Apa Oh V. 2(2)

A more useful tool: Using economic evaluation to make

better decisions

APA Ohio State Planning ConferenceOctober 20 & 21, 2011

Dayton, Ohio

Agenda

• Introductions

• Definitions and goals of economic evaluations• Economic evaluation vs. economic impact• How to do an economic evaluation• Discussion

3

What are the Goals of Economic Evaluations?

• Because all resources are scarce, economic evaluations help us…..– Identify which initiatives will offer the greatest

value• “Value” means providing the most benefit per

unit of cost• This is the so-called “efficiency criterion.”

Guarantees that the greatest aggregate benefit will be provided for that cost.

– But efficiency has little to say about equity; that is, about how the total benefits get distributed

4

Why do we need to do economic evaluations of our planning and policy choices?

• We know different choices have different costs and impacts, but we don’t know what those are automatically. – As a result, we can either figure it out or

“guesstimate”– Too often, we guesstimate – and get it

wrong.

• We can use economic evaluation methods as a tool kit to make our real-world decision making better.

5

How does this differ from an economic impact study?

• Economic IMPACT: essentially a political tool– Problem of counting indirect

impacts: who’s right?– Not always balanced against cost– Most useful for persuasion– Losing effectiveness due to overuse?

6

Therefore….

• Use an economic impact study if are trying to persuade people to do something.

• Use an economic evaluation if you are trying to find answers and make decisions.

Types of economic evaluation

8

The Groundwork for Economic Evaluations

• Burden of Cost – What impact will changing a policy have on people?– Cost may be economic, social, or psychological cost to

individual, organization, or community.

– Example: changing how a city service is delivered.– Who will be directly impacted?– Who will be impacted as a result of the first set of

impacts?– What will it require for the impacted people/orgs to make

up the difference, or avoid a loss of quality?

9

What Makes a Decision Hard?

• Uncertainty – Do we know what’s really

going on?– Do we know what the best

approach will be?– Do we know if this has

worked in the past?– What happens if we get it

wrong?

• Urgency

– How soon will it happen?– How soon do we have to

deal with it?

• Conflicting objectives

– Cars vs. pedestrians?– Commercial or residential?– Quality of service vs. range

of access?

10

Cost-Effectiveness Analysis

• Compares the relative costs and outcomes (effects) of two or more courses of action– Costs are measured in monetary units and outcomes are

measured in effectiveness, such as deaths averted , park use increased or VMT improved.

– Typically the CEA is expressed in terms of a ratio where the denominator (bottom) is the impact from the potential action, and the numerator (top) is the cost.

_COST_OUTCOME

COST:OUTCOME

11

Cost-Effectiveness Analysis Example

Building a new park

Cost: expenditure required to build the park according to design.

Benefit factors: number of people expected to use the park, impact on health of users (health costs, weight loss, health care costs).

You’ll note that this is the tricky part….can be done, but make assumptions clear.

12

Cost-Benefit Analysis

• Cost-benefit analysis (CBA) differs from CEA and CUA in that benefits are measured in monetary units, just like costs

• This feature allows one to compare widely different types of interventions using CBA, since all benefits and costs are expressed in monetary units.

• But with public policy issues, converting outcomes to cash isn’t always a good idea.

13

Conceptual Underpinnings of CEA

• Unlimited wants

• Finite resources

• Inevitability of choices

• Balance of benefits and costs

• Need for formal analysis

14

Solving the Resource Allocation Problem

• The approach: constrained optimization• Two ways of framing the problem:

What allocation of resources will achieve the greatest possible gains without exceeding the volume of available resources?

What allocation of resources will minimize our expenditures, subject to the requirement that we achieve at least some target level of benefit?

15

Finding the Cost-Effective Solution

• Identify mutually-exclusive interventions– Eliminate strongly and weakly dominated interventions

– Select the most effective program with a CE ratio less than or equal to some threshold value

threshold value = decision maker’s willingness to pay for a unit of benefit

More costly, less effective than some competing alternative

+

+

-

Calculateincremental

CE ratio

Calculateincremental

CE ratio

Dominant

Dominated

-

Incremental Cost-Effectiveness Ratios

Incr

emen

tal

Effec

tiven

ess

Incremental Cost

16

17

Incremental CE Ratios

• The incremental CE ratio comparing P2 to P1:

Cost of P2 – Cost of P1

Effect of P2 –Effect of P1

This tells you how much more you would have to spend for each additional unit of benefit you would gain if you choose a more costly, more effective

program rather than a less costly, less effective program

18

Using an Incremental CE Ratio to Guide Choice

Incremental Effect

Incr

emen

tal C

ost

slope = l

“BAD”[CE ratio >l]

“GOOD”[CE ratio <l]

19

Importance of Incremental Analysis

Tx Cost Effect

A $100,000 10 LM

B $110,000 10.001 LM

20

Importance of Incremental Analysis

Tx Cost Effect CE

A $100,000 10 LM $10,000/LM

B $110,000 10.001 LM $10,999/LM

Importance of Incremental Analysis

Tx Cost Effect CE Incremental CE

A $100,000 10 LM $10,000/LM —

B $110,000 10.001 LM $10,999/LM $10,000,000/LM

Importance of Incremental Analysis

Tx Cost Effect CE Incremental CE

A $100,000 10 LM $10,000/LM —

B $110,000 10.001 LM $10,999/LM $10,000,000/LM

Right way to

calculate CE ratios!

Wrong way to

calculate CE ratios!

23

Incremental Analysis With Multiple Options

• Example: Image three different street segments, and three different paving options:– Option 1: Pot Hole Repair– Option 2: Walnut Street, chip and seal– Option 3: Walnut Street, grind and repave– Option 4: Main Street, chip and seal– Option 5: Main Street, grind and repave– Option 6: George Parkway, chip and seal – Option 7: George Parkway, grind and repave– Option 8: Vine Street, chip and seal– Option 9: Vine Street, grind and repave

• Since each road carries a different volume of traffic, and since each paving option has a different lifespan, each has different costs and different length of benefit.

24

Costs and Effects

Strategy Cost ($) Life span/traffic volume ratio

1 40,288 39.08

2 44,786 42.56

3 45,944 43.04

4 47,046 43.01

5 48,596 43.60

6 54,628 43.20

7 56,812 43.83

8 58,082 43.80

9 61,119 44.62

25

Costs and Effects

9

87

6

543

2

140,000

45,000

50,000

55,000

60,000

65,000

39 40 41 42 43 44 45

Life span/traffic volume ratio

Cost

($)

26

9

7

5

32

140,000

45,000

50,000

55,000

60,000

65,000

39 40 41 42 43 44 45

Life span/traffic volume ratio

Cost

($)

Eliminate Strongly Dominated Strategies

27

Calculate CE Ratios

Strategy Cost ($) Effect Incremental CE ratio

1 40,288 39.08 —

2 44,786 42.56 (44,786-40,288)/(42.56-39.08) = 1,293

3 45,944 43.04 (45,944-44,786)/(43.04-42.56) = 2,413

5 48,596 43.60 (48,596-45,944)/(43.60-43.04) = 4,736

7 56,812 43.83 (56,812-48,596)/(43.83-43.60) = 35,722

9 61,119 44.62 (61,119-56,812)/(44.62-43.83) = 5,452

28

Identify and Eliminate Weakly Dominated Strategies

Strat. Cost ($) Effect Incremental CE ratio

1 40,288 39.08 —

2 44,786 42.56 (44,786-40,288)/(42.56-39.08) = 1,293

3 45,944 43.04 (45,944-44,786)/(43.04-42.56) = 2,413

5 48,596 43.60 (48,596-45,944)/(43.60-43.04) = 4,736

7 56,812 43.83 (56,812-48,596)/(43.83-43.60) = 35,722

9 61,119 44.62 (61,119-56,812)/(44.62-43.83) = 5,452

29

Identify and Eliminate Weakly Dominated Strategies

9

7

5

32

140,000

45,000

50,000

55,000

60,000

65,000

39 40 41 42 43 44 45

Life span/traffic volume ratio

Cost

($)

30

Recalculate CE Ratios

Strategy Cost ($) Effect Incremental CE ratio

1 40,288 39.08 —

2 44,786 42.56 (44,786-40,288)/(42.56-39.08) = 1,293

3 45,944 43.04 (45,944-44,786)/(43.04-42.56) = 2,413

5 48,596 43.60 (48,596-45,944)/(43.60-43.04) = 4,736

9 61,119 44.62 (61,119- 48,596)/(44.62-43.60) = 12,277

31

Recalculate CE Ratios

9

5

3

140,000

45,000

50,000

55,000

60,000

65,000

39 40 41 42 43 44 45

Effect

Cost

($)

2

32

Which Strategy to Use?

• Move down the list of undominated programs until the CE ratio of the next program is “too high”

• What is threshold CE ratio?– No correct answer

How to set up an economic evaluation

34

Basic Steps in an Economic Evaluation

1. Define the problem

2. Identify the relevant alternatives

3. Estimate expected costs

4. Estimate expected effects

5. Compare expected costs and effects

6. Time preferences & discounting

7. Analyze uncertainty

8. Sources of data

35

Define the Problem

• Choice of perspective is particularly important– Whose costs are considered relevant?

• Possible perspectives– Local government– Departments– Nonprofits– Subsets of residents– Businesses – Residents

• Whatever you choose, make it clear

36

Identify the Relevant Alternatives

• Possible comparators– “Best” alternative– Most widely used alternative– “Do nothing”

• Appropriate comparator may vary by location

• Potential for bias

37

Sources of Data

• Big challenge in public policy– We don’t measure well– Think through what you need – Gather local data if possible– Make clear what you are assuming

38

• Associate resource utilization with outcomes and events

• Associate costs with resource utilization

Calculate Expected Costs

39

Direct Costs

• Construction & Development•Operating Expenses• Staff/staff time•Advising/Consulting

40

Indirect Costs

• Impact on others– Cash costs for service

replacement/augmentation

– Time costs to users– Impacts on other

programs’ capacity to meet objectives

41

Compare Costs and Effects

• Identify strongly and weakly dominated strategies

• Calculate incremental cost-effectiveness ratios

42

Time Preference and Discounting

• Prefer to receive benefits now, defer costs to future

• Present value of X occurring T years from now

= X/(1+r)T

Time preference

• Discount costs and benefits at same rate

• Most widely used rate: 3% (typically in range 0%-10%)

Common practice

43

Analyze Uncertainty

• Perform sensitivity analysis– Helps to gauge the

reliability and robustness of the results

– Particularly important for public policy, given data problems

• Identify ranges of values for model inputs

• Recalculate economic evaluation using these different values

44

45

Thank you !

Peter Mallow, AICPResearch Associate – S2 Statistical SolutionsPhD Student – University of [email protected]

Della Rucker, AICP, CEcDPrincipal – Wise Economy Workshopwww.wiseeconomy.comDella.rucker@wiseeconomy.com513.288-6613Twitter: @dellaruckerFacebook: Della Rucker Aicp CecdAlso on Google + and LinkedIn