Experience you can trust. Market Transformation Programs: What they are. How to evaluate them. Kansas Corporation Commission Mitchell Rosenberg, Vice President.

Experience you can trust.

Market Transformation Programs: What they are. How to evaluate them.

Kansas Corporation CommissionMitchell Rosenberg, Vice President

Topeka, Kansas

March 26, 2008

2

Overview

Market Transformation: Definitions and Reality Importance of MT Concepts in Energy Efficiency

Program Design and Evaluation Key Challenges in MT Evaluation Steps in Meeting MT Evaluation Challenges:

examples from actual programs & studies Lessons Learned

3

Definition of Market Transformation

Market Effect: “a change in the structure or functioning of a market or the behavior of participants in a market that results from one or more energy efficiency program efforts.”

– Typically, these efforts are designed to increase the adoption of energy-efficient products, services, or practices, and are causally related to market interventions.

Market Transformation: Market effects that persist once supporting programs are terminated

4

MT References the Product Life Cycle

0%

10%

20%

30%

40%

50%

60%

70%

80%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Year

Ma

rk

et

Sh

are

Introduction Early Acceptance Take Off Maturity

SUPPLY CHAIN

Smaller competitors or new entrants producing No standardization Very high prices Performance problems Distribution mostly in non-standard channels Limited product lines Little marketing support

1 – 2 larger competitors enter Product standards develop Prices high relative to standard Improved performance Limited distribution Limited model line build out Some marketing support

Most large competitors offer product Product standards adopted voluntarily Prices are higher but in line with standard models Product lines built out Good distribution Manu. Marketing support

All major competitors in market Mandatory standards Prices approach those of standard Producers compete on price and features Distribution via all channels Retailer marketing support

CUSTOMERS

Low level of awareness Skepticism of product claims Only early adopters buying

More customers aware Continued skepticism of product claims Small expansion of market beyond early adopters

Strong demand in advanced segments Some demand in all segments

Strong demand in all segments

5

Policy & Programs to Accelerate MT

0%

10%

20%

30%

40%

50%

60%

70%

80%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Year

Ma

rk

et

Sh

are

Introduction Early Acceptance Take Off Maturity

SUPPLY CHAIN ORIENTED Government lab R&D Sponsored corporate R&D Technology road mapping Mediate technology standard setting Development of performance metrics and testing protocols

Vendor technical and sales training Co-advertising Vendor merchandising support Development & promotion of voluntary product efficiency standards Product testing

Vendor technical and sales training Co-advertising Vendor merchandising support Upstream product subsidies Initiate consideration of higher product standards Develop common service specifications

Mandatory codes and standards Promulgate higher voluntary standards

DEMAND ORIENTED

Purchase of prototypes or early models Develop and publicize case studies of applications

Bulk purchase Customer education Rebate programs General EE public relations

Customer education Rebate programs General EE public relations

Continued customer education Rebate programs for higher efficiency units only

6

EE Programs do help transform markets: The case of electronic ballasts

0

10

20

30

40

50

60

70

80

90

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

An

nu

al B

allast

Sh

ipm

en

ts, M

illio

ns o

f U

nit

s

0

50

100

150

200

250

300

An

nu

al V

alu

e o

f C

on

str

ucti

on

, 1992 D

ollars, $ B

il.

Electronic

Magnetic

Non-residentialConstructionValue

R&D: LBL & CA utilities

Government Procurement

DSM Programs @ $50m/yr

DSM Programs @ $150m/year

Programs Underway

7

The case of resource-efficient clothes washers

1998 2000 2002 2004 2006

Local Programs in Effect

12 50 70 100+ 90

ENERGY STAR & Other Federal Events

1st ENERGY

STAR

specification (1997)

DOE announces new min. standard

National promotions initiated

Federal min standard increased

(2007) Fed min standard and ENERGY STAR

increased

Manufacturers producing ENERGY STAR models

8 14 17 21 24

Number of ENERGY STAR models

18 35 84 125 212*

* Includes only those that meet the revised 2007 specification.

0%

5%

10%

15%

20%

25%

30%

35%

40%

1996 1998 1999 2000 2001 2002 2003 2004 2005 2006

Mar

ket

Sh

are

8

Why MT Concepts are Important in Program Design & Evaluation Program designs must be matched to current stage of

development to be cost effective Harnessing supply side motivations enhances cost

effectiveness High GHG reduction goals

– Need continuous pipeline of energy-efficient products, services, design practices

– Need to extend adoption of established products to all relevant market segments

In some cases, market research provides most fruitful approach to NTG evaluation

9

Key Challenges in MT Evaluation Effects occur over a long time frame

– Events of interest span decades– Program planning & evaluation cycles typically 3 years or less

Multiple program sponsors and policy makers are involved

– Often coordinate activities among jurisdictions– How tease out effects of one state’s programs?

Data are difficult to obtain

– Manufacturer shipment data often proprietary– Distributors, contractors, retailers, designers tend to be small,

scattered, disorganized

10

Steps in Addressing the Challenges of MT Evaluation Market Assessment/Program Logic Model Program History Development: Market Presence

Indicators Selection of analysis objectives and strategy (ies) Appraisal of results/Adjustments for next rounds

Close parallels to steps in program design and revision

11

Program Logic Model: Home Performance

12

Market Assessment/Program Theory Key Objectives

– Location on the product life cycle

– Identification of key market actors & segments, motivations, barriers

– Assess data availability for program management and evaluation

– Get program planners, implementers, regulators on same page re:

short & long term goals and expectations Timing: Should begin in program design phase

Resources: Avoid over-investment

– Secondary data & reports useful

– Selected local primary research often very useful

13

Program History/Market Presence Important to build reference for market effects and

attribution analysis Should include national as well as local events &

trends over time Data to capture

– Program activities: marketing, training, incentives– Participation: customer and vendor, numbers and

characteristics of participants and measures, timing, region

– Efficiency levels supported v. baseline practice

14

Example: EVT clothes washer programs

1998 2000 2002 2004

NATIONAL PROGRAMS AND MARKETS

Local Programs in Effect 12 50 70 100+

ENERGY STAR & Other Federal Events

1st ENERGY STAR

specification DOE announces new min. standard

National promotions initiated

Federal min standard increased

Manufacturers producing ENERGY STAR models

8 14 17 21

Number of ENERGY STAR models

18 35 84 125

ENERGY STAR Market Share 6.2% 9.3% 16.0% 27.2%

LOCAL PROGRAMS AND MARKETS

ENERGY STAR Market Share 24.9% 27.3% 35.7% 49.6%

Number of Partic. Retailers 40 58 70 75

Number of Incentives Paid 1,950 2,476 2,699 4,129

Number of ENERGY STAR units sold without rebates

7 938 1,401 2,131

15

Analysis Strategies Market Effects Indicators

– Market share: sales, prevalence of practices, fleet efficiency indicators (e.g. mean EERs, MPG)

– Price trends– Changes in codes and standards

Attribution Analysis

– Cross sectional– Time Series– Self-reporting– Historical

Best to attempt multiple indicators and attribution analysis approaches

16

Net Effects of Vermont CW programs:Cross Sectional & Historical Approach

Results of cross-sectional modeling

– Dependent variable: State level Energy Star CW market share

– Independent variables:

Program indicators

Customer Demographics & Energy Prices

Change indicator

Model Estimated Mkt. Share % of E Star Sales

Attributed to Program

Year Actual

Market Share No

Program No Program

+ No Cum. Effect Program

Only Program +

Cum Effect Variable

2001 22.6% 12.9% n/a 34% n/a

2003 52.6% 46.8% 43.4% 7% 26%

2004 48.9% 46.1% 43.4% 6% 12%

17

VT clothes washer programs: What’s happening to local program influence

Methods

– ‘No-program Area’ market shares rising faster than market areas

Market conditions

– Impending federal minimum standard changes

– Profitable product for manufacturers

Fair representation of local program effects?

1999 2000 2001 2002 2003 2004

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

No Program

US Average

Program

VT

18

CA commercial lighting programs: presence in the market

Sponsor Electronic Ballasts Rebated US Utility Ballasts Rebated

US EB Shipments

mm Units mm Units

As % of Program

Area EB Sales

As % of all

US EB Shipments mm Units

As % of US

EB Shipments

1992 13.3 0.9 90% 7% 9.4 71%

1993 24.5 1.2 90% 5% 13.2 54%

1994 24.6 2.0 75% 8% 12.2 50%

1995 32.9 1.9 67% 6% 11.1 34%

1996 30.3 1.3 57% 4% 10.5 35%

Total 125.6 7.3 68% 6% 56.4 45%

19

General Challenges from C&I Programs Little available market share data Complex ‘measures’ Greater customer market segmentation Multiple levels of supply chain with direct influence on

project-level equipment selection and design Multiple decision makers and criteria in one customer

Generally greater reliance on ‘weight of evidence’ and self-reported approaches than in residential.

20

Other Potential Approaches to Attribution or Baseline Development Diffusion modeling Delphi or other expert judging Conjoint analysis of preferences for efficient

substitutes for current products

21

Lessons Learned

Generate as detailed a story as possible

– Know the history of the program and its relatives– Know the history of the market and technology– Know what other programs are doing– Develop the program logic with local stakeholders

Know the available data resources

– Sampling, contents, collection methods– Know what others have done

Design data collection to the attribution strategy or strategies

22

Lessons Learned: Maximizing data opportunities

Quality of Indicators

– Accuracy, face validity, bias (lack of same) Replicability (ability to support historical or time

series analysis)

– Sample frame: captures full population, updated regularly, documented compilation

– Sampling: keep it kosher and document it Comparability (supports cross-sectional analysis)

– Same definitions as data collected elsewhere – Capitalize on channels to reach national markets

23

The Bigger Picture

Play well with others

– Markets addressed by local programs are national and international

– Many program operators are heavily involved already – get to know them

– Initiatives under way to procure sales & shipment data Independent program influence?

– Many local programs already coordinate operations, or are developed in explicit reference to each other (e.g.) codes & standards

– Why try to tease apart effects?