Empirical Data on Settlement of Weather Sensitive Loads Josh Bode, M.P.P. ERCOT Demand Side Working Group Austin, TX September 20, 2012.

Empirical Data on Settlement of Empirical Data on Settlement of Weather Sensitive LoadsWeather Sensitive Loads

Josh Bode, M.P.P.Josh Bode, M.P.P.

ERCOT Demand Side Working GroupERCOT Demand Side Working GroupAustin, TXAustin, TX

September 20, 2012September 20, 2012

Presentation Overview

Why is settlement of weather sensitive loads an issue?

Testing accuracy of settlement methods

Empirical results

Using smart meter data and control groups for evaluation

Page 2

Baselines are a tool to estimate demand reductions

Measuring demand reductions is an entirely different task than measuring power production Power production is metered and thus is measured directly.

Demand reductions cannot be metered. They must be estimated by indirect approaches.

In principle, the reduction is simply the difference between electricity use with and without the load curtailment However, it is not possible to directly observe or meter what

electricity use would have been in the absence of the curtailment – the counterfactual.

Instead, the counterfactual must be estimated.

Page 3

The accuracy of baseline estimates for large C&I customers has been studied multiple times

Page 4

KEMA Baseline Analysis for CEC

WG2 Baseline accuracy analysis

(Quantum)

WG2 Report Baseline Accuracy Analysis

(Quantum)

LBNL Study (proxy events)

Ontario Power Authority Study

(FSC)

California Aggregator and

DBP Evaluations(CAEC)

Highly volatile load customer study

(CAEC)

ISO-NE Baseline Study

(KEMA)

PJM Baseline Study

(KEMA)

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

California Aggregator Programs

(FSC)

Settlement of reductions from weather sensitive loads like AC has been studied less

Weather sensitive loads have demonstrated the ability to support multiple grid functions

4.8 million residential AC units and more than 500,000 water heaters in the U.S. have load control devices

Recent technological innovations enable aggregation and real time visibility of small scale loads

Many load control devices now include over and under frequency relays, providing an automated fail-safe mechanism that is synchronized with the grid

Page 5

Visibility of loads has been tested

Because of the sheer number of AC units, it is not practical to monitor all data points

Real time monitoring of AC units is expensive, even for a sample

Page 6

Feeder Load

AC end use sample

Estimated loads for population

Load control programs have shown the ability to provide contingency reserves

Page 7

0 15 30 45 60 75 90 105 120 135 150

Computer to modem Modem to devices

0%

20%

40%

60%

80%

100%

120%

140%

-4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Nor

mal

ized

Avg

. Cus

tom

er k

W

Time Since Event Start

Average

Fast start up timeFast ramp up to full resource capability

Highly granular dispatch is possible – it is possible to dispatch all or some of the resources in a specific area

No one has tested the built-in under frequency relays which provide a failsafe mechanism, do not rely on central dispatch and should respond even faster

Customers that were curtailed 68-71 over the summer report same comfort and frequency of events as customers that were curtailed once

Water heaters have demonstrated the ability to follow regulation signals

Page 8

Graph is from PJM pilot (Joe Callis) The initial test was for a unit and has since expanded to wider scale testing

However, these loads are highly variable

Average Hourly Residential AC Loads by Temperature

Page 9

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

0:00 3:00 6:00 9:00 12:00 15:00 18:00 21:00 0:00

Avg

Dem

and

Per A

C U

nit

(kW

)

Hour Ending

0 CDD

1-2 CDD

3-4 CDD

5-6 CDD

7-8 CDD

9-10 CDD

11-12 CDD

13-14 CDD

15-16 CDD

17-18 CDD

19-20 CDD

21-22 CDD

The fact that some loads are very weather sensitive does not mean they are unpredictable

Testing the accuracy of settlement methods

Page 10

We tested 11 different settlement alternatives for baselines for short curtailments

Type of

EstimatorMethod No. Calculation

Data SourceIndividual

ACAggregate

ACFeeder

HouseData

Within- subject

estimators

Day-matching baseline

1 10-in-10 with a 20% in-day adjustment cap X X X X2 10-in-10 without an in-day adjustment cap X X X X3 Top 3 in 10 without an in-day adjustment cap X X X X

Weather- matching baseline

4 Profile selected based on daily maximum temperature without an in-day adjustment cap X X X X

Regression

5 Treatment variables and no day or hourly lags or leads X X X X6 Treatment variables with a day lag X X X X7 Treatment variables with hourly lags and leads X X X X8 No treatment variables but use of hourly lags and leads X X X X

Between- subject

estimators

Random assignment of load control operations

9 Comparison of Means X X

10 Difference-in-differences X X

Pre-calculated load reduction estimate tables 11

Multiply the number of AC units in each geographic location by the corresponding estimate of demand reductions per AC unit for the corresponding area, hour of day, and temperature bin.

X X

Page 11

To test accuracy, one needs to know the correct values

Page 12

Because the demand reductions values are artificially introduced and

known, we can determine the accuracy of each baseline alternative

For the tables and approaches that relied on control groups, we used a split sample approach

1. Randomly split data into two groups

2. Simulate the reduction in one group

3. Use the second group to produce the counterfactual or baseline

4. Calculate impacts and store

5. Repeat 100 times

Page 13

REPEAT 100 TIMES

GROUP AUSAGE WITH AC CONTROL

▪ Unperturbed electricty use patterns are known

▪ Substract simulated impacts for each event are from actual load, producing and estimate of load with DR

GROUP BUSED TO PROVIDE COUNTERFACTUAL

▪ Unperturbed electricty use patterns are known

▪ Unperturbed electricty use patterns are used to estimate what Group A's electricity use would have been absent AC control - the counterfactual.

Randomly select sample from relevant data

source

Randomly assign to Groups A and B

CALCULATE IMPACT ESTIMATE

▪ Simple comparison of means: difference between average unperturbed electricity use ofGroup B minus average of load with DR for Group A.

▪ Regression: Explain difference in electricity use pattern to produce a more precise estimate of the impact

STORE RESULTS(Actual and estimated impacts)

There are two key issues in assessing accuracy – bias and goodness of fit

Type of Metric Metric Description

Bias Mean Percentage Error (MPE)

The mean percentage error (MPE) indicates the percentage by which the measurement, on average, tends to over or underestimate the true demand reduction.

Goodness-of-Fit

Mean Absolute Percentage Error (MAPE)

The mean absolute percentage error (MAPE) is a measure of the relative magnitude of errors across event days, regardless of positive or negative direction. It is normalized allowing comparison of results across different data sources.

CV(RMSE) This metric is similar to MAPE except that it penalizes large errors more than small ones

Page 14

0.00

0.20

0.40

0.60

0.80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Redu

ction

per

AC

unit

(kW

)

Simulated Reduction Baseline estimate

Avg. actual reduction Avg. Baseline Estimate

Limitations of analysis

The analysis of settlement approaches focuses on ancillary services Short term reductions to stabilize the grid

Usually triggered by generation or transmission outages and sometimes unexpected changes in wind or loads

Not always in the hottest hours

The errors in the demand reduction estimates depend on the magnitude of the reduction signal The estimates were based on 50% standard cycling

Air conditioner use is lower in California than in Texas

The direction of the findings likely hold up but the magnitude of the errors will be different

Page 15

How does the magnitude of the demand reduction signal affects accuracy? Example

Metric Customer A Customer B

Baseline Estimate 294 kW 294 kW

True Reference load 300 kW 300 kW

Load with DR 270 kW 225 kW

Demand reduction estimate 24 kW (294-270) 69 kW (294-225)

True demand reduction 30 kW (10%) 75 kW (25%)

% Error -20% -8%

Page 16

100

150

200

250

300

350

12 -

1 AM

3 -4

AM

6 -7

AM

9 -1

0 AM

12 -

1 PM

3 -4

PM

6 -7

PM

9 -1

0 PM

kW

True Load without DR Baseline

Actual Event Day Load

100

150

200

250

300

350

12 -

1 AM

3 -4

AM

6 -7

AM

9 -1

0 AM

12 -

1 PM

3 -4

PM

6 -7

PM

9 -1

0 PM

kW

True Load without DR Baseline

Actual Event Day Load

The customers are nearly identical, but the estimation error differ because of they reduce ad

different amount of demand

Empirical Results

Page 17

What is the value of more complex approaches?

In each case, we compare results to the most simple approach – the pre-calculated load reduction tables

We present the results for within-subject and control group approaches separately

We present the bias and goodness of fit metrics separately

All graphs used the same scale

Page 18

Matching and regression approaches with individual AC data

Page 19

182% 224% 272% 164% 26% 29% 30% 35% 39% 41% 44%0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

Typi

cal %

Err

or (C

VRM

SE)

-136%

94% 96% 55% 80% 2% 6%

-3% -6%

0% 7%

-20%

-10%

0%

10%

20%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

BIAS

(MPE

)

Matching and regression approaches with aggregated AC data

Page 20

126% 50% 43% 35% 26% 29% 31% 22% 39% 41% 44%0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

Typi

cal %

Err

or (C

VRM

SE)

-105%

14% 9%

-3%

3% 5% 15%

-2% -6%

0% 7%

-20%

-10%

0%

10%

20%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

BIAS

(MPE

)

Matching and regression approaches with whole house data

Page 21

188% 92% 57% 40% 25% 27% 28% 30% 39% 41% 44%0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

Typi

cal %

Err

or (C

VRM

SE)

-123% -36% -11% -6%

0% 12% 14%

-8% -6%

0% 7%

-20%

-10%

0%

10%

20%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

BIAS

(MPE

)

Matching and regression approaches with feeder data

Page 22

2984% 2010% 1227% 1530% 317% 362% 401% 726% 39% 41% 44%0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

Typi

cal %

Err

or (C

VRM

SE)

-1810% -526% -203% -463% -89% -140% -98% -72% -6%

0% 7%

-20%

-10%

0%

10%

20%

1 2 3 4 5 6 7 8 5% 50% 95%

Day-matching Weather-Matching

Baseline Methods Regression Methods Impact Estimate Tables

BIAS

(MPE

)

why are feeder results so inaccurate? Example

Feeder characteristics 2,672 accounts on feeder 266 AC load control accounts (10% of

feeder) 292 AC controllable AC units Likely includes commercial Penetration higher than 90% of feeders

Event day characteristics August 24, 2010, max temp 103 F Simulated event period 12:00-14:00

• AC load per unit 0.63 kW• Load Impact 35%• Controllable AC load 183.3 kW (0.63

kW per unit x 292 AC units)• Feeder Impact 64 kW • Actual load without DR 7772 kW• Simulated load with DR 7708 kW

Percent impact on feeder 0.83%!!!

Page 23

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

0 3 6 9 12 15 18 21 24

Feed

er L

oad

(kW

)

Hour Ending

Baseline (+1% bias)

Actual Load without DR (SCADA)Simulated Load with DR

Smart AC participant house loadControllable AC Load

Control group methods with AC end use data(500 control group, 500 treatment)

Page 24

23% 34% 55% 20% 30% 43% 39% 41% 44%0%

20%

40%

60%

80%

100%

5% Median 95% 5% Median 95% 5% Median 95%

Comparison of Means Diff-in-diff Impact Estimate Tables

Between Subjects Impact Estimate Tables

Typi

cal %

Err

or (C

VRM

SE)

-32% -2%

39%

-16%

0% 14%

-6%

0% 7%

-30%

-20%

-10%

0%

10%

20%

30%

5% Median 95% 5% Median 95% 5% Median 95%

Comparison of Means Diff-in-diff Impact Estimate Tables

BIAS

(MPE

)

Control group methods with whole house data (2,000 control group, 2,000 treatment)

Page 25

17% 24% 52% 15% 20% 27% 39% 41% 44%0%

20%

40%

60%

80%

100%

5% Median 95% 5% Median 95% 5% Median 95%

Comparison of Means Diff-in-diff Impact Estimate Tables

Typi

cal %

Err

or (C

VRM

SE)

-41% -2%

32%

-9% -1%

10%

-6%

0% 7%

-30%

-20%

-10%

0%

10%

20%

30%

5% Median 95% 5% Median 95% 5% Median 95%

Comparison of Means Diff-in-diff Impact Estimate Tables

Between Subjects Impact Estimate Tables

BIAS

(MPE

)

Implications of study Don’t rely on feeder data for settlement

Day matching baselines are the least accurate approach with weather sensitive loads

Day-matching baselines are not well suited for measuring demand reductions from highly weather sensitive loads

More granular meters do not necessarily increase the accuracy of demand reduction measurement because measuring demand reduction is fundamentally different

Complex methods provide limited improvement

Pre-calculated load reduction tables can produce results that on average are correct, but may err for individual days, especially if they are cooler

Methods with control groups and large sample sizes perform best

Page 26

Using Smart Meter Data and Control Groups for

Evaluation

Page 27

Impact estimate tables are not developed in a vacuum

They should be based on a history of results

Ideally this includes systematic testing of load control devices under different condition and with different operation and control strategies

The estimates from the operations underlying the tables need to be unbiased and, ideally, precise

The more data points, the better the results

One may need to account for changes in the customer mix, if relevant

Page 28

With large samples and random assignment, estimation error is virtually eliminated

Actual example for 2011 PG&E SmartAC evaluation

Wide availability of smart meter data and individually addressable devices are a pre-requisite

Page 29

Randomly assign population into 10 groups

For each test event, one group was activated and the other 9 were held as a control groups

For a few events, we tested different operation strategies side-by-side

It enables side by side testing of different operation strategies

Page 30

It also enables side by side testing of different control strategies

Page 31

By using control groups and short events, once can get a substantial history of results

In the PG&E study, each customer only experienced one event, but we obtained results from 7 days, including 3 with side by side testing

It is reasonable to call up to 10 events per customers, especially if the curtailments are short (e.g. 1-2 hrs)

This can yield results under 100 different curtailments to inform the impact estimate tables

Page 32

Page 33

For any questions, feel free to contact

Josh Bode, M.P.P.

Freeman, Sullivan & Co.101 Montgomery Street 15th Floor, San Francisco, CA 94104

[email protected]