Performance Metrics for Weatherization David Carroll State WAP Evaluations ACI Home Performance Conference April 30, 2014.

Performance Metrics for Weatherization

David Carroll

State WAP Evaluations

ACI Home Performance Conference

April 30, 2014

What is Performance Measurement?

2

Step 1 - Goals• What are your goals?

– Are you trying to serve a certain population?– Are you trying to affect …

• Energy use?

• Energy costs?

• Payment problems?

• Emissions?

• Health problems?

– How will you know when you are successful?

3

Step 2 – Measurement• Overall Research

– Who are you serving?– How are you serving them?– What are they saving?– What other benefits are you delivering?

• Comparative Research– What approach maximizes energy, cost,

payment, emissions, health impacts?

4

Step 3 - Assessment

• Assessment– What are your goals for your program?– Does your performance match your goals?– How can you better align your performance

with your goals?

5

Step 4 – Improvement

• Strategies– Changing who you serve

• Targeting

• Partnerships

– Changing what you deliver• Assessments

• Measures

6

Step 4 - Improvement

• Strategies (continued)– Changing your performance

• Comparative analysis

• Mentoring / partnerships

– Changing funding available• Leveraging

• Advocacy

7

A Comprehensive Approach to Performance Measurement

8

Benchmark

• Use data on ….– Population / Census Data (ACS)– Recipients / LIHEAP or Energy Assistance– WX Clients / Demographics, Housing Units,

Baseline Conditions

9

Observe

• Observations of ….– Intake– Audits– Service Delivery– Inspections

10

Measure

• Collect data on ….– Installed Measures– Post-WX Conditions– Pre/Post Energy Usage– Client Outcomes – Payments, Housing, Health

11

Analyze

• Study ….– Gross changes in weather-normalized usage– Net changes using a comparison group– Variations in outcomes by key client, building,

and program factors

12

Assessment

• Investigate…– Why are some subgroups seeing high impacts?– Why are some subgroups seeing low impacts?– What strategies might be tested to better align

outcomes with goals?

13

Who are you Serving? Example of using Census and LIHEAP

data as a Benchmark

14

Minnesota Households - 2010

About one in four Minnesota households were income-eligible for the LIHEAP Program in 2010.

15

Eligibility Group Number of Households Percent of Households

Not Income Eligible 1,564,187 75%

Income Eligible 527,359 25%

All Households 2,091,546 100%

Source: 2010 American Community Survey

Minnesota LIHEAP and WAP Recipients in 2010

About 38% of income-eligible households received LIHEAP and 2% received WAP.

National averages are 17% and 0.5%, respectively.

16

Received BenefitsLIHEAP

RecipientsPercent of

HouseholdsWAP

RecipientsPercent of

Households

Yes 199,969 38% 11,207 2%

No 327,390 62% 516,152 98%

Income-Eligible Households

527,359 100% 527,359 100%

Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database

Housing Unit Type

LIHEAP is less likely to serve households in Large Multi-Family buildings. WAP serves very few households in Multi-Family buildings of any type.

17

Housing Unit Type Low-Income LIHEAP Recipients WAP Recipients

Single Family 52% 59% 80%

Mobile Home 5% 11% 14%

Small Multi-Family 8% 7% 3%

Large Multi-Family 35% 23% 3%

Total 100% 100% 100%

Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database

Housing Unit Type

LIHEAP is less likely to serve households in Large Multi-Family buildings. WAP serves very few households in Multi-Family buildings of any type.

18

Housing Unit Type Low-Income LIHEAP Recipients WAP Recipients

Single Family 52% 59% 80%

Mobile Home 5% 11% 14%

Small Multi-Family 8% 7% 3%

Large Multi-Family 35% 23% 3%

Total 100% 100% 100%

Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database

Ownership Status

LIHEAP serves both owners and renters. WAP is less likely to serve renters.

19

Ownership Status Low-Income LIHEAP Recipients WAP Recipients

Owner 46% 44% 84%

Renter/Other 54% 56% 16%

Total 100% 100% 100%

Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database

Ownership Status – Single Family Units

Even if we control for housing unit type, WAP is more likely to serve owners than renters.

20

Ownership Status Low-Income LIHEAP Recipients WAP Recipients

Owner 75% 60% 87%

Renter/Other 25% 40% 13%

Total 100% 100% 100%

Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database

Ownership Status of WAP Participants – Single Family Units

Control for housing unit type reduces that variation in service to renters by substate area.

21

Ownership Status

State NE NW Central Metro SE SW

Owner 87% 93% 94% 89% 80% 88% 88%

Renter/ Other

13% 7% 6% 11% 20% 12% 12%

Total 100% 100% 100% 100% 100% 100% 100%

Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database

How are you serving them? Example of furnishing details for

the state, region, and agency.

22

23

Table 4 PY 2010 Clients in Single Family Homes

Housing Unit Characteristics by Region and Agency

Statistic STATE Central Region

Lake Wobegone

Number of Units 8,863 1,292 100

Housing Unit

Median Heated Space 1,340 1,235 1,200

Mean Heated Space 1,451 1,336 1,350

% One Story 40% 46% 49%

Housing Vintage

% pre 1940 38% 38% NC

% 1940-1969 33% 27% NC

% 1970 or later 29% 35% NC

PreWX Status

Mean CFM 50 2,337 2,163 2,855 NC=Not Collected

24

Table 7 PY 2010 Clients in Single Family Homes

Air Sealing and Shell Measures by Region and Agency

Statistic STATE Central Region

Lake Wobegone

Number of Units 8,863 1,292 100

Air Sealing

< 500 CFM Reduction 54% 58% 50%

500 - < 1000 CFM Reduction 23% 20% 15%

1000+ CFM Reduction 19% 17% 32%

No Information 5% 5% 3%

Any bypass sealing or caulking 81% 75% 82%

Attic Insulation

% Installed 86% 83% 90%

Wall Insulation

% Installed 36% 36% 39%

Other Insulation

% Floor insulation 5% 7% 6%

% Rim/band joist insulation 19% 41% 15%

% Foundation insulation 6% 6% 10%

Mean CFM reduction 607 548 840

% Installed Mechanical Ventilation 46% 57% 55%

How much energy are you saving and how does that vary by important program factors?

25

Table 1WAP Energy Impacts for Single Family Site-Built Homes

Gross and Net Gas Savings by End Use (therms/year)

Group/Breakout # HomesGas Use Pre-WAP

Gas Use Post-WAP

Gross Savings Net Savings % of Pre

Total Use 937 983 845 138 (±10)

130 (±11) 13.2% (±1.1%) Comparison 1,295 929 921 8 (±4)

Heating Use 937 775 658 117 (±9)

100 (±11) 13.0% (±1.4%)

Comparison 1,295 741 724 16 (±6)

Baseload Use 937 208 187 21 (±7)

30 (±8) 14.2% (±17)

Comparison 1,295 188 196 -8 (±5)

Table 3WAP Energy Impacts for Single Family Site-Built Homes

Gas Savings for Homes with Natural Gas Main HeatBy Measure Combination (therms/year)

Group/Breakout # HomesGas Use Pre-

WAP Net Savings % of Pre

No Major Measures 202 877 37 (±15) 4.2% (±1.7%)

One Major Measure 298 957 121 (±17) 12.7% (±1.8%)

Two Major Measures 211 1,003 162 (±20) 16.1% (±2.0%)

Three Major Measures 115 1,111 236 (±29) 21.2% (±2.6%)

All Four Major Measures 33 1,179 382 (±71) 32.4% (±6.1%)

 Table 4

WAP Energy Impacts for Single Family Site-Built HomesNet Gas Savings for Natural Gas Main Heat by Pre-Weatherization

Gas Usage (therms/year)

Pre-WAP Gas Use (therms/yr)

Major Measures # Homes

Gas Use Pre-WAP Net Savings % of Pre

All Clients 1.4 937 983 130 (±10) 13.2% (±1.1%)

<750 th/yr. 1.1 245 640 64 (±12) 10.0% (±1.9%)

750-1000 1.3 296 880 105 (±14) 12.0% (±1.6%)

1000-1250 1.6 226 1,097 142 (±22) 12.9% (±2.0%)

1250-1500 1.6 101 1,355 219 (±42) 16.2% (±3.1%)

>=1500 th/yr. 2.0 69 1,731 269 (±65) 15.6% (±3.7%)

Table 12Gas Impact Results by Agency for Gas Heated

Single Family Site-Built Homes

Agency IDGas Use Pre-WAP Net Savings % of Pre

# of Measures

A 1,077 187 (± 21) 17.3% (±1.9) 2.2B 992 122 (± 32) 12.3 % (±3.2) 0.9C 1,028 119 (± 40) 11.6% (±3.9) 1.2D 948 118 (± 19) 12.4% (±2.0) 1.3E 1,012 113 (± 64) 11.1% (±6.3) 0.7F 937 109 (± 44) 11.6% (±4.7) 1.2G 945 107 (± 50) 11.3% (±5.3) 0.9H 875 94 (± 26) 10.7% (±3.0) 0.7I 929 94 (± 41) 10.1% (±4.4) 1.1J 889 58 (± 27) 6.5% (±3.0) 0.5

Total 983 130 (±11) 13.2% (±0.7) 1.4

Make sure you consider other program impacts.

30

Table 26Emissions Benefits

Estimated Value of Avoided Emissions per Home for Single Family Site-Built Homes (2013 Dollars)

Pollutant First Year Lifetime

CO2 Equivalent $53.65 $1,009.63

NOX $23.45 $432.11

PM2.5 $2.49 $42.85

SO2 $44.48 $684.39

VOC $0.41 $8.25

Total $124.49 $2,177.22

31

Dwelling Quality Indicators

32

IndicatorPre-Audit Incidence

Post-WX Incidence

Percent Change

Home sometimes at unhealthy temperature

20% 9% -55%

Observed standing water in home

33% 19% -42%

Frequent mildew odor or musty smell

29% 16% -45%

Home somewhat or very infested with insects

25% 17% -32%

Home has CO monitor 41% 74% +80%

Affordability Indicators

33

IndicatorPre-Audit Incidence

Post-WX Incidence

Change

Got disconnect notice 45% 33% -26%

Had natural gas, electricity, or bulk fuel service terminated

18% 9% -50%

Went without food to pay energy bills

34% 23% -33%

Went without food in the last four weeks

9% 6% -33%

Used high interest loan to pay energy bills

20% 12% -40%

Contact

34

David Carroll, [email protected]

APPRISE32 Nassau Street, Suite 200Princeton, NJ 08540