Report Group D – D01.01 Workplan for 2018-19 Industrial, Agricultural, and Large Commercial (IALC) Custom Impact Evaluation Submitted to California Public Utilities Commission 505 Van Ness Avenue San Francisco, CA 94102 Submitted by SBW Consulting, Inc. 2820 Northup Way, Suite 230 Bellevue, WA 98004 In association with BuildingMetrics Inc. Energy350 Opinion Dynamics Ridge & Associates Tierra Resource Consultants June, 7 2019
50
Embed
Workplan for 2018-19 IALC Custom Impact Evaluation · Total 2,449 140 51% 55% 5% 229 95 82% 5% . Workplan for 2018-19 IALC Custom Impact Evaluation SBW Consulting, Inc. 2 Methodology
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Report Group D – D01.01
Workplan for 2018-19 Industrial, Agricultural, and Large Commercial (IALC) Custom Impact Evaluation
Submitted to California Public Utilities Commission 505 Van Ness Avenue San Francisco, CA 94102
Submitted by SBW Consulting, Inc. 2820 Northup Way, Suite 230 Bellevue, WA 98004
In association with BuildingMetrics Inc.
Energy350
Opinion Dynamics
Ridge & Associates
Tierra Resource Consultants
June, 7 2019
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. ii
Acknowledgements
We want to acknowledge the critical contributions from CPUC Energy Division staff, who
guided the development of the workplan and provided many improvements and clarifications
regarding CPUC policies and relevant practices.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. iii
Table of Contents
List of Figures .............................................................................................. v
List of Tables .............................................................................................. vi
List of Acronyms and Abbreviations ............................................................ vii
Table 6: 2018 IALC Sample Design Summarized by Program Administrator – Electric Frame ........................................................................................................................................ 15
Table 7: 2018 IALC Sample Design Summarized by Program Administrator – Gas Frame ........... 15
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. vii
List of Acronyms and Abbreviations
Acronym/ Abbreviation
Definition
AQMD Air quality management districts
CATI Computer-assisted telephone interview
CMPA Custom measure and project archive
CPUC California Public Utilities Commission
EM&V Evaluation, measurement and verification
ER Early retirement
ESPI Efficiency savings and performance incentive
EUL Effective useful life – The average time over which an energy efficiency measure results in energy savings, including the effects of equipment failure, removal, and cessation of use.
HOPPs High opportunity projects and programs
IALC Industrial, agricultural, large commercial
IOUs Investor owned utilities
IPMVP International performance measurement and verification protocol
ISP Industry standard practice
kW Kilowatts
kWh Kilowatt-hours
M&V Measurement and verification
MMBtu Millions of British thermal units
NMEC Normalized metered energy consumption – This is a means by which savings are quantified at the meter level through a regression approach that normalizes energy use by independent variables such
as weather, production, etc.
NTGR Net-to-gross ratio
PA Program administrators
PG&E Pacific Gas and Electric
POE Preponderance of evidence
RUL Remaining useful life – The number of remaining years that an item, component, or system is estimated to be able to function
SCE Southern California Edison
SCG Southern California Gas
SDG&E San Diego Gas and Electric
SEM Strategic energy management
QA/QC Quality assurance and quality control
UTC Use and technology categories
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 1
Executive Summary
In this workplan, we—the SBW team—describe how we will complete the ex post evaluation of
custom-project claims that were reported by program administrators (PAs) for 2018 and 2019.
We have designed these evaluations to provide reliable estimates of life-cycle net kWh, kW and
therm savings for the custom-project portfolio. A separate workplan covers the portion these
savings that are attributable to the Industrial Strategic Energy Management (SEM) program.
For each PA we have defined specific sample domains based on the project’s classification using
three factors: sector (industrial, agricultural, and large commercial), custom-project type (new
construction or retrofit), and fuel type (electric or gas). Within each of these sample domains we
will estimate the life-cycle net savings and the life-cycle costs and benefits for each PA. We will
also provide actionable recommendations to improve future ex post evaluation methods and to
improve the PA’s savings claims for future custom projects.
Sample Design
Table 1 shows our sample design for estimating gross savings for the 2018 evaluation. Our
design represents the electric and gas savings claims for each PA with a sampling precision that
exceeds ±10% at 90% confidence in all cases. The sampled projects account for 51% of reported
kWh savings and 82% of reported therm savings. Additional projects will be selected within
each sample domain to meet our sample size objectives for net-to-gross ratio (NTGR) surveys.
We will use a similar design for 2019, and we will select projects at the end of each calendar
quarter. This will provide additional time for data collection and allow us to conduct the NTGR
surveys with decision-makers closer to the times that they made their decisions to proceed with
their projects. It will also enable us to adjust the sampling domains to adequately represent new
types of projects, such as commercial normalized metered energy consumption (NMEC), which
may become more important over the course of the evaluation.
Table 1: Sample Design for Gross Savings Evaluation of 2018 Custom Projects
Electric Sample Design Gas Sample Design
PA Project Count
Sample Count
Percent kWh
Savings Sampled
Percent kW
Savings Sampled
Relative Precision
Project Count
Sample Count
Percent Savings
Sampled
Relative Precision
MCE 40 14 70% 53% 10% NA NA NA NA
PG&E 2,004 56 47% 53% 7% 89 46 80% 7%
SCE 291 41 59% 56% 6% NA NA NA NA
SCG NA NA NA NA NA 91 26 87% 8%
SDG&E 114 29 63% 72% 9% 49 23 82% 10%
Total 2,449 140 51% 55% 5% 229 95 82% 5%
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 2
Methodology
We will conduct the 2018 and 2019 evaluations in two phases. In the first phase, we will review
all reported custom-project claims and make corrections as necessary to bring them into
compliance with California Public Utilities Commission (CPUC) policies. We will then
estimate the adjusted gross and net savings due to the corrections in this review. In the second
phase, we will conduct site inspections, interview decision-makers with NTGR surveys, and
review submetering and other types of data collection to support our estimates of ex post gross
and net savings for the sampled projects. Using the sample results, we will then estimate the ex
post life-cycle net savings and the cost-effectiveness for each sample domain for every PA.
Schedule
Figure 1 shows our schedule for completing this work.
Figure 1: Schedule for Ex Post Evalution of 2018 and 2019 Custom Projects
Year:
Deliverable Task / Subtask Due date Qtr: 1 2 3 4 1 2 3 4 1 2 3 4
Evaluation work plan . 2
Draft release - ED review 20-May-19 2
Draft release - stakeholder review 7-Jun-19 2
Final posted 6-Jul-19 2
Plan update Apr-20 2
Data collection and sampling plan Jul-19 . 2
2018 project data requests Jun-19 2
2019 project data requests Jul-19 2 2 2
Gross savings estimates
2018 reviews, data collection, analysis Dec-19 . .
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 3
1 Background
In this section, we summarize our understanding of the findings from prior evaluations of
custom projects, issues raised in recent energy savings performance incentive (ESPI) memos,
and relevant CPUC policies and guidance.
1.1 Prior Evaluations
Custom programs were evaluated in 2013, 2014, and 2015.1 The results of these evaluations
were remarkably similar. Gross energy-realization rates and NTGRs consistently hovered
between 0.5 and 0.6. The reasons cited for the relatively low realization rates and NTGRs
remained consistent across the studies. The primary reasons include:
� Errors in the calculation methods of the PAs
� Inappropriate baseline specifications
� Savings claims for ineligible projects
� Use of default operating conditions instead of forecast conditions or observed operating
conditions
Recommendations across the studies paralleled these reasons and included:
� The IOUs should improve calculation methods and protocols to increase the accuracy of
savings estimates.
� The IOUs should improve the documentation and reporting of the effective useful life (EUL)
of projects and the remaining useful life (RUL) of early-retirement (ER) projects.
� Improve the quality control of project operating conditions, ex ante baseline determinations,
savings calculations, and eligibility rules.
� Adjust project savings based on post-installation inspections and measurement and
verification (M&V).
� PAs should consider adopting program-implementation procedures and features designed to
increase program influence.
1.2 Issues Identified in ESPI Performance Memos
Issues identified during ex-ante review (EAR) were documented in the ESPI memos. Recurring
themes across PAs and program years included:
1 The 2013 study separated custom retrofit and nonresidential new construction (NRNC) projects as separate reports, while the 2014 and 2015 reports combined custom retrofit and NRNC projects.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 4
� Calculation tools. Issues with the EnergyPro2 software used to calculate savings for the
Savings by Design program were identified on projects submitted by PG&E, SCE and
SDG&E. Similar issues occurred across multiple program cycles.
� Quality assurance/quality control (QA/QC). Persistent issues with internal due diligence
review were identified across all PAs.
� Measure life. Persistent problems with the correct assignment of measure effective useful
life and remaining useful life for accelerated replacement projects were identified. EULs for
retrofit add-on measures that exceed the RUL of the host equipment were also identified.
� Program influence documentation. Inadequate documentation of program influence was a
persistent problem across all PAs.
� M&V. EAR dispositions required additional M&V on multiple projects and measure types
for which measured performance data was needed to estimate savings. Measures requiring
additional M&V included: guest room energy management systems, non-DEER lighting
projects, and first-time projects with little performance history.
� Problematic measures. Issues were identified with savings calculations for several measure
types, requiring improvements to calculations, input parameters, or documentation. These
measures included: chiller optimization, air compressor plant projects, oil pumping,
monitoring-based commissioning, variable air volume conversions, and waste water
treatment plants.
1.3 CPUC Policies and Guidance
An important goal of this evaluation will be to determine whether the utility-reported energy
savings for custom projects in 2018 and 2019 conform to applicable CPUC policies and
guidance. As described in section 3, we will review all reported claims to determine whether
they conform to the policies and guidance provided the following documents:
� CPUC Energy Efficiency Policy and Procedures Manual (v. 5 or the most recent version) 3
� Utility Statewide Custom Policy and Procedures manual4
� Statewide Savings by Design Policy and Procedures manual5
� PA-specific policy and procedures manuals
� Industry standard-practice (ISP) studies completed and adopted before or during 2017
2 EnergyPro is building energy performance modeling software, sold by EnergySoft LLC, which is approved by the
California Energy Commission for demonstrating performance compliance with the nonresidential provisions of the California Building Energy Efficiency Standards.
3 CPUC. 2013. Energy Efficiency Policy Manual Version 5.0 For Post-2012 Programs. San Francisco, CA: CPUC. 4 PG&E, SCE, SoCalGas, SDG&E. 2018. 2018 Statewide Customized Offering Procedures Manual for Business. San
Francisco, CA: CPUC. 5 CPUC. 2018. Savings by Design 2018 Participant Handbook. San Francisco, CA: CPUC.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 5
� Title 20 and 24 requirements in place when projects were permitted.
� CPUC policy papers and state-government memos addressing topics such as non-IOU fuel
sources and the requirements for preponderance of evidence for EUL/RUL.
� CPUC resolution E-4818 regarding assignment of project baselines
� Review dispositions of custom projects issued by CPUC staff
� NTG ratio and EUL/RUL support tables downloaded from READI v.2.5.1 available at
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 6
2 Evaluation Objectives
This workplan will meet the following objectives for the 2018 and 2019 program years:
� Gross Savings. We will estimate life-cycle gross kWh, kW and therm savings for the
custom-project portfolio, excluding the portion attributable to the Industrial SEM program,
which is covered under a separate workplan. As described in section 4, for each PA we will
define sampling domains based on three factors: sector (industrial, agricultural, and large
commercial), custom-project type (new construction or retrofit), and fuel type (electric or
gas). Within each of these sampling domains we will estimate gross savings for each PA.
During 2019, other domains may be added if new types of projects, such as commercial
NMEC, become an important part of the custom-project portfolio.
� Net Savings. We will estimate a net-to-gross ratio that can be multiplied by gross savings to
calculate the net savings attributable to custom projects within each domain. Market effects
will be accounted for by adding a market effects adder of 0.05 to all NTGR values, as
directed by the CPUC.
� Cost-Effectiveness. We will estimate life-cycle costs and benefits for each domain, for each
PA, and for the overall custom-project portfolio, based on incremental costs and the most
appropriate effective useful life and load shape.6 For early-retirement efficiency measures,
we will account for remaining useful life when we estimate life-cycle costs and benefits.
� Reproducible Results. We will document all our primary data collection efforts, modelling,
and data processing procedures to ensure that our results are transparent and can be
reproduced by other parties.
� Recommendations. We will develop actionable recommendations designed to improve the
PA savings claims for future custom projects and to improve future ex post evaluation
methods.
6 Load shapes will be selected from those shapes supported by the CPUC cost-effectiveness tool.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 7
3 Review of Claims Database
This section describes the first phase of our evaluation in which we will review all savings claims
associated with custom measures reported by the PAs for 2018 and 2019 (The second phase of
our evaluation will focus on a sample of projects as described in sections 4 and 5.) In this phase,
we will examine all savings claims to determine whether these claims conform to CPUC policies
and guidance. The following subsections describe the criteria we will use to determine if savings
claims comply with CPUC polices and guidance, and the procedures for correcting claims that
do not comply.
3.1 Data Requests
Our review will be limited to claims data that is submitted by the PAs to CEDARS and other
information that it is practical for the PAs to provide for all their claims. Other information
may include clarification of installation dates, indication of hard-to-reach customers, values for
missing NTG or EUL IDs, second baseline savings, and clarification of measure descriptions.
3.1.1 Review Project Claims
We will review these claims to identify whether additional information is required to support a
review of compliance with CPUC policies. We will develop requests for additional information
as identified in this review.
3.1.2 Request Supplemental Data
Our data requests will address any critical missing information that could reasonably be
provided. Based on a recent review of claims for 2017, we anticipate requesting the following
information from the PAs.
� Installation Date. For claims with installation dates prior to the 2018 or 2019 program
years, we will request verification that the claims were delayed due the time needed to
complete M&V after installation. For claims with dates occurring after the program year, we
will request explanations regarding why the claims were not held and claimed for that
program year.
� Hard to Reach. For claims associated with hard-to-reach customers, we will request
identifying documentation.
� DEER NTG and EUL IDs. For claims missing either the DEER NTG or EUL ID, we will
request those ID values.
� Second Baseline Savings. For claims whose measure application type indicates an early-
retirement measure, we will request any missing savings for the second baseline.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 8
� Measure Description. For claims with ambiguous or vague descriptions, we will request
clarification.
3.2 Review Reported Claims
3.2.1 Correct Claims Based on Supplemental Data
We will create an updated version of the reported claims based on the responses to the
supplemental data requests by either correcting values provided to CEDARS or adding elements
of data not included in CEDARS.
3.2.2 Claim Year
The CPUC directed the PAs to include only savings for measures that were installed in the same
program year for which they are reporting. In 2017, the CPUC provided further guidance to the
PAs that savings applied towards ESPI payments will only be counted for measures that were
installed in the program year for which ESPI payments are being requested. The guidance
requires the PAs to identify the installation year for all projects that are included in their claims
for that program year. Additionally, the CPUC directed PAs to identify claims with justifiable
exceptions to the rule regarding installation-dates. Unless sufficient documentation supports an
appropriate exception, any claimed savings occurring outside of the program year will be set to
zero, effectively removing them from the claim.
3.2.3 Classification Based on Measure Description
We will examine information in the measure description field of the claims database and correct
errors in the values of the following fields: use category, use subcategory, tech group, and tech
type.7 These corrections will result in updates to the values for NTGR, EUL, and RUL.
Measure descriptions will be used to establish each claim’s NTG, EUL, RUL, and measure
application type. We will match the measure descriptions for each claim with the DEER ex ante
NTG, EUL and RUL values8 that were adopted for use in 2018 or 2019. This will enable us to
map measure descriptions to values for specific use category, use subcategory, tech group, and
tech type. These are referred to collectively as UTCs, which is an abbreviation for use and
technology categories9.
7 The CPUC’s Remote Ex Ante Database Interface (READI) utilizes use category and use subcategory to define how or
where a technology is used. All technologies that are referenced in measure definitions are further organized into tech groups and tech-types (i.e., a subcategory of tech group).
8 NTGR and Full EUL and RUL support tables downloaded from READI v.2.5.1 available at: http://www.deeresources.com/index.php/deer-versions/readi
9 Tables defining each of the UTCs (use category, use subcategory, tech group and tech type) are available from READI v.2.5.1 available at: http://www.deeresources.com/index.php/deer-versions/readi
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 9
To ensure a claim is correctly categorized according to its UTCs, we will develop a keyword list
of names and terms that are common to each measure description use category. This will
provide the highest level of classification for claims and will be used to further match a claim to
an accepted value in the ex ante database. We will develop the keyword list from the measure
description from each claim and the use categories contained in the ex ante database10 measure
list that was adopted for use in 2018 and 2019. Table 2 gives examples of keyword matches.
Table 2: Example of Keyword List
Ex Ante Database Use Category Claims Database Measure Description Keywords
Based on the distribution of projects and savings in the 2018 program year shown in Table 3
above, we developed a sample design shown in Table 4 and Table 5 below. For the estimated
gross savings within each domain, we have targeted the 90% level of confidence with a relative
precision of plus or minus 15% or better while for the estimated gross savings within each PA,
we have targeted the 90% level of confidence with a relative precision of plus or minus 10% or
better.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 14
Within each domain, we selected a stratified random sample13 that minimizes the required
sample size to achieve the targeted levels of confidence and precision. Such a design can provide
the same level of confidence and precision as a simple random sample but with a smaller sample
size. Furthermore, each domain has a certainty stratum containing the largest savings project(s)
in that domain. The projects in the certainty strata have a probability of selection of 1 and,
therefore, no sampling error. Also, most domains have an exclusion stratum containing the
smallest savers which are excluded from the sampled strata. Our use of exclusion and certainty
strata further reduced sample sizes, while maintaining the confidence and precision targets.
Table 4 and Table 5 present the target sample sizes for each domain in the electric and gas
frames, respectively. For domains with a small number of projects, i.e., fewer than 10, we
selected all projects. For example, for the domain defined as PGE-SBD/NC-AG, which has
only two projects, both projects were selected. For these domains, as in the case of the certainty
strata, there is, by definition, no sampling error. Fifteen projects are in common between the
electric and gas samples. Each of these 15 projects count toward the required sample size in
their respective fuel frames.
Table 4: 2018 IALC Sample Design – Electric Frame
Sample Domain Total Project Count
Sample Size
Total LifeCycle Net Savings
(GWh)
Sampled Lifecycle Net
Savings (GWh)
Percent of Savings in
Sample
Target Relative Precision PA
Impact Type
Sector
MCE Retrofit COM 40 14 9,997 7,018 70% 10%
PGE SBD/NC AG 2 2 17,945 17,945 100% 0%
COM 20 5 117,955 86,138 73% 9%
IND 2 2 2,903 2,903 100% 0%
Retrofit AG 134 12 127,178 69,312 54% 14%
COM 1,664 24 450,481 83,685 19% 10%
IND 182 11 394,325 264,690 67% 15%
SCE SBD/NC COM 18 8 58,009 41,082 71% 13%
Retrofit AG 20 10 16,467 11,941 73% 8%
COM 207 14 205,372 129,651 63% 9%
IND 46 9 72,097 23,603 33% 12%
SDGE SBD/NC COM 42 15 68,124 41,695 61% 13%
Retrofit COM 66 8 81,411 48,988 60% 14%
IND 6 6 9,758 9,758 100% 0%
Total 2,449 140 1,632,023 838,409 51%
13 Stratified random sampling is a method of sampling that involves the division of a population or subpopulation
(domain) into smaller sub-groups known as strata. In stratified random sampling, the strata are formed based on
members' shared attributes or characteristics such as income or educational attainment, or in our case, savings of similar size. See Chapter 5 of Sampling Techniques, Cochran, (1977).
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 15
Table 5: 2018 IALC Sample Design – Gas Frame
Sample Domain Total Project Count
Sample Size
Total Lifecycle Net Savings
(kBtu)
Sampled Lifecycle Net
Savings (kBtu)
Percent of Savings in
Sample
Target Relative Precision PA
Impact Type
Sector
PGE SBD/NC AG 1 1 6 6 100% 0%
COM 15 8 1,773 1,509 85% 14%
IND 2 2 34 34 100% 0%
Retrofit AG 4 4 1,584 1,584 100% 0%
COM 53 23 9,855 8,434 86% 10%
IND 14 8 9,627 6,720 70% 13%
SCG Retrofit AG 1 1 3,049 3,049 100% 0%
COM 67 13 3,439 2,271 66% 13%
IND 23 12 6,334 5,835 92% 15%
SDGE SBD/NC COM 29 16 1,056 979 93% 15%
Retrofit COM 19 6 3,010 2,334 78% 12%
IND 1 1 90 90 100% 0%
Total 229 95 39,857 32,844 82%
As shown in Table 4 and Table 5, this sample design of 140 electric projects and 95 gas projects,
for a total sample size of 235, touches 51% of claimed electric savings and 82% of claimed gas
savings. Table 6 summarizes the sample design by PA for the electric frame followed by Table 7
for the gas frame.
Table 6: 2018 IALC Sample Design Summarized by Program Administrator – Electric Frame
PA Project Count
Sample Count
Total Lifecycle Net Savings
Sample Lifecycle Net Savings Percent Savings
Sampled Sample Relative
Precision GWh GW GWh GW
MCE 40 14 9,997 1 7,018 1 70% 10%
PGE 2,004 56 1,110,788 136 524,672 72 47% 7%
SCE 291 41 351,944 38 206,277 21 59% 6%
SDGE 114 29 159,293 18 100,442 13 63% 9%
Total 2,449 140 1,632,023 193 838,409 106 51% 5%
Table 7: 2018 IALC Sample Design Summarized by Program Administrator – Gas Frame
PA Project Count
Sample Count
Lifecycle Net Savings (kBtu)
Sample Lifecycle Net Savings (kBtu)
Percent Savings Sampled
Sample Relative Precision
PGE 89 46 22,878,405 18,286,192 80% 7%
SCG 91 26 12,822,855 11,155,400 87% 8%
SDGE 49 23 4,155,980 3,402,752 82% 10%
Total 229 95 39,857,240 32,844,345 82% 5%
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 16
We will oversample in each of the sampled strata in all domains to ensure that we have
replacements in case customer representatives from our primary sample selections are not
responsive. We will carefully monitor sample disposition throughout the evaluation to ensure
we are closely following the sample design to produce reliable results.
4.2 Net Savings Sample for 2018
We will conduct NTGR survey interviews with customer decision-makers from the projects in
the sample described above. We will randomly select additional projects within each domain
and strata to complete a total of 285 NTGR survey interviews.
4.3 Gross and Net Savings Sample for 2019
We will select samples after the end of each calendar quarter to represent the custom projects
installed in 2019. By sampling each quarter, we will
1. Allow more time for data collection, which will be especially beneficial for those projects
requiring sub-metering of affected systems or equipment.
2. Conduct the NTGR survey with decision-makers closer to the time that the decision was
made to proceed with each project.
3. Adjust the sampling domains to adequately represent new types of projects, such as
commercial NMEC, which may become more important over the course of the year.
Our gross and net samples sizes for the year in total, will again be 235 and 285 respectively, and
we expect to use a sample design like the one we described for 2018. We will develop a
quarterly forecast of completed projects by domain, based on the annual pattern of project
installation during 2017 and 2018, to guide the allocation of the sample across the quarters. We
expect somewhat lower sampling precision from this quarterly design, but we believe that
disadvantage is outweighed by the benefits listed above.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 17
5 Ex Post Evaluation Methods
This section summarizes our ex post gross and net evaluation methods. This section also
describes our approach to analyzing the measure costs, lifetimes, and load shapes needed to
assess cost-effectiveness in section 6.
5.1 Data Requests
We will issue a series of data requests to the PAs for the files and other information needed to
complete ex post evaluation of the gross and net samples for 2018 and 2019.
5.1.1 Obtain Project Files
We will request that the PAs upload all files for the sampled projects to the evaluation,
measurement and verification (EM&V) platform on DEERresources.info. We will retrieve the
uploaded files and move them to secure storage that is accessible to our analysts. We will review
the project files to determine whether they are missing any critical data, models, or
documentation.
5.1.2 Request Missing Data, Models, or Documents
We will use the EM&V platform to request any missing critical data, models, or documentation
for sampled projects. Based on our recent experience reviewing a sample of 2017 projects and
on our experience with previous ex post evaluations, we anticipate that these requests may
include:
� Working calculators. Spreadsheet models or hourly model input files.
� Fuel switching justification. If a retrofit project results in an energy consumption shift from
electricity to gas or vice versa, then the three-prong test14 is required to assess eligibility for
switching fuel.
� On-site generation analysis. Any on-site generation, such as cogeneration or behind the
meter solar arrays at the site, requires analysis that considers the marginal change in IOU-
supplied fuels resulting from the project.
� Justification of early retirement and remaining useful life. Early-retirement projects
require evidence that the existing equipment is functional and could continue to be used for
the claimed remaining useful life.
� Evidence of program influence. Eligible projects require evidence that project initiation was
influenced by the program.
14 Energy Efficiency Policy Manual Version 5, page 24, Cost Effectiveness Rule XV.10
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 18
� Permitting. If permits are required for measure installation, the customer/installer must
obtain such permits.
� Contacts. Names and contact information for decision-makers or other parties familiar with
the systems, equipment or practices affected by the project.
� M&V data. Trend logs, interval data, billing energy use data, or other metering records that
were obtained by the program during the baseline or post-installation periods. This would
include any test results associated with commissioning the affected systems or equipment.
� Affected systems, equipment or practices. Clarification of the characteristics of the systems,
equipment, or practices that were affected by the project, including both baseline and post-
installation conditions.
� Costs. Invoices or other related cost information used to determine the measure cost.
5.2 Gross Savings Evaluation
This section describes how we will develop site-specific measurement and verification plans for
each sampled project, including special circumstances for new construction and site-specific
NMEC projects. This section also discusses how our team will determine ex post conditions and
use those findings to re-estimate project gross savings.
5.2.1 Prepare Site-Specific M&V Plans
We will develop site-specific M&V plans to guide the NTGR interviews, site visits, data
collection efforts, and the procedures for estimating savings. These M&V plans will also serve as
the basis for the final project-specific M&V reports. To speed the development of these
individualized M&V plans, we’ll begin by preparing a template for a general M&V plan for
common project types that we can then customize to accommodate more unique project types.
We will submit draft and final templates to CPUC staff for review and approval.
These general templates will be well-suited to projects that feature custom capital retrofit and
operational measures in the industrial, agricultural and large commercial sectors. M&V plans
for such projects must deal with questions regarding normal and early replacement, and they
must apply International Performance Measurement and Verification Protocol (IPMVP)
Options A, B or D in their approach. However, custom projects in the following two subject
areas also have unique attributes that must be dealt with somewhat differently:
� New construction capital measures (Savings by Design)
� Site-specific NMEC, which includes high opportunity projects and programs (HOPPs).
We discuss the special approaches for these subject areas in section 5.2.1.2 and section 5.2.1.3,
respectively. To avoid scattering these topics across many subsections, each section instead
treats these topics comprehensively by fully describing the specific issues and unique situations
that affect M&V planning, data collection, and analysis.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 19
5.2.1.1 Features of All M&V Plans
Once we have selected a sample we will use the approved template to create customized M&V
plans for each sampled project. These customized M&V plans will serve as roadmaps guiding
the NTGR interviews, site visits, data collection, and the procedures for estimating savings. Our
team will also standardize the processes for plan development, technical support, and quality
assurance to ensure that each site-specific M&V plan conforms to this workplan, meets quality
control standards, and remains consistent with other similar plans that have already been
approved.
After we review the project documentation and fully understand the nature of the implemented
measure(s), one of the first steps in M&V planning will be to recruit the customer for the M&V
effort. Doing so dramatically minimizes the chance that customers will not cooperate and
therefore minimizes the likelihood of any time wasted on planning efforts. Early customer
contact also enables the person in charge of leading the M&V plan to collect key logistical
information to inform the approach to the plan by better understanding the type of data that will
be available for the project and whether onsite data logging is appropriate. We will use
information obtained from the customer contacts to refine the M&V plans prior to the site visits.
During our conversations with customers we will also verify that the equipment has been
commissioned and that it is operating as defined in the project documentation. If any significant
changes have occurred since the time that the measure was implemented, we will make
appropriate adjustments to our M&V planning.
A critical aspect of the M&V planning will be to ensure that the approaches we follow are
consistent with the guidelines developed for the ex ante reviews of the projects. The rulebook
developed for the ex ante reviews contains key guidance, rulings, and decisions, and it will be an
important resource for developing M&V plans.
In a similar vein, we will coordinate with the team working on the Group A deemed evaluation
to ensure that both groups are applying consistent approaches for evaluating HVAC and lighting
measures. We will also work with the Group A team to identify any customers that both groups
have sampled so we can coordinate and minimize customer contacts. The best time to
coordinate these efforts will be immediately after the sample is drawn.
We will develop a comprehensive training regimen to ensure that all field staff are
knowledgeable about M&V planning, field procedures, and the proper use of the data collection
forms. This will ensure effective data collection and compliance with the evaluation protocols
and CPUC guidance. We will also develop a handbook and provide real-time support while our
staff are in the field to ensure their questions and issues can be addressed promptly.
Each plan will address the key questions listed below.
Is the project eligible?
Before developing a full-fledged customized M&V plan for a project, we will request project
documentation as described in section 5.1.2 and perform an initial project review to assess
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 20
project eligibility. While this review will not include an assessment of free-ridership, which will
be covered elsewhere, it will look for relevant information, such as projects that were completed
outside of the program year without corresponding M&V and fuel switching that fails the three-
prong test. If we determine that a project is ineligible, we avoid spending time developing
unnecessary plan elements.
How will we analyze savings?
As a starting place for developing our customized M&V plans, we will review the methodology
used for analyzing the ex ante savings estimates. In many cases, we will pursue a similar
calculation strategy using updated data to determine ex post savings. However, depending upon
the available data, we may determine that an alternative evaluation methodology will yield
more accurate savings results. In some cases, alternative evaluation methodologies may also be
dictated by data availability at the time of the evaluation.
We will identify any parameters with substantial uncertainty and determine whether contact
with the customer or vendor is needed to mitigate this uncertainty. We will also determine
whether it is sufficient to rely on a phone call or email, or whether it is necessary to conduct an
onsite visit for inspections, measurements, or metering.
Implicit in our review and corresponding planning for data collection is establishing whether the
project falls into the basic or enhanced rigor level.15 These indicate the level of resources and
effort required to achieve reliable evaluation savings estimates for that project. Necessary
resources depend on project characteristics, which can be difficult to predict in advance. For
example, a large, complex project may be easy to evaluate because it has well-documented
models and abundant data from before and after the measure installation. Conversely, a small,
simple project may require more resources to evaluate because it lacks sufficient data and it
requires a new analysis approach that must be developed from scratch.
What data should we collect?
Project M&V leads will develop data collection forms and survey instruments that are
customized according to the site-specific M&V plan. For example, commercial building projects
may require onsite survey forms to verify and/or update the building characteristics and
operations data that are used to drive building simulations, while industrial process projects may
require customized data collection forms to suit the unique needs of the evaluation. In instances
where a project contains early-retirement-measure application types, there may be claims for
dual baselines, and the plan will necessarily address estimating savings for both.
Typical data collected for projects will include:
� Verification of installation and operation of the specified equipment
15 Rigor level is defined for program evaluations in the 2006 California Energy Efficiency Evaluation Protocols. For
engineering models, these protocols call out IPMVP Option A (Partial Retrofit Isolation) for basic rigor, and Option B
(Retrofit Isolation) for enhanced rigor. Our approach is consistent with this general philosophy, applied to specific projects and measures.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 21
� Operational conditions, including load, hours of use, and seasonal variations
� Power measurements of equipment that are taken over a representative range of operating
conditions and a period of time that is long enough to establish normal operational
parameters with a high degree of certainty
� Trend data from onsite monitoring systems or building management systems that show
equipment operation
� Production data, if equipment operation is directly related to production
Often the ex ante project files contain data collected from the site, including facility trend data,
baseline conditions, and operation immediately after project installation. In addition to the ex
ante data, we have found it is best to collect updated trend data from the site since conditions
sometimes change, either in response to operational changes or seasonal variations. The best
estimates of long-term energy use can often be obtained by combining data from the ex ante
analysis with data collected during an evaluation site visit.
What about net savings?
Projects will also be sampled separately in the net savings evaluation described in section 0.
That section describes our plans for customizing or adapting survey scripts and interview guides
to estimate free-ridership for these projects.
5.2.1.2 New Construction
This evaluation will account for any unique aspects of new construction projects within the site-
specific M&V planning described above. New construction projects under Savings by Design
(SBD) cover a wide variety of sectors and industries, including green-field new construction, as
well as major expansions and renovations to existing buildings and industrial processes. SBD
offers both a whole building approach and a systems approach to commercial new construction
projects.
Incentives are paid based on energy savings calculated using an approved calculation tool.16 The
whole building approach uses compliance software to estimate the compliance margin (savings
above code) and calculate incentives. Projects permitted under Title 24, versions 2013 and
newer, use compliance tools based on software from CBECC-Com: California Building Energy
Code Compliance (for commercial/nonresidential buildings), which uses EnergyPlus as the
simulation engine. Previous rounds of the IALC evaluation sampled buildings that were
permitted under the 2008 standards that used an EnergyPro simulation engine driven by
DOE2.1E software. Buildings to be sampled for this evaluation likely received permits under
Title 24, versions 2013 and newer. Our evaluation data collection and software interface
processes will adapt to this new simulation environment.
16 One example of a SBD tool is EnergyPro Version 7, a software package approved by the California Energy
Commission for use with the whole building approach. For the systems approach, programs typically use simplified calculation approaches and tools, such as SimCalc.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 22
In addition to Title 20 and Title 24, our M&V plans will consider regulations governing the
minimum code-compliant baselines for new construction projects. These include emission limits
imposed on industrial process equipment by the various air quality management districts
(AQMD) in the state. We will develop data requirements for documenting compliance with
AQMD regulations. Because high-efficiency industrial equipment may be needed to comply
with AQMD regulations, this may limit the energy savings options that are available beyond
code.
As part of our sampled project review, we will assess any calculation tools applicable to the
savings estimates for industrial process measures. The CPUC currently maintains the
Calculation Tool Archive. Our team will assess new processes and tools encountered during the
evaluation and add them to the archive as appropriate.
Some of the risks, challenges, and barriers associated with evaluating SBD projects include:
� Dealing with a particularly wide variety of commercial and industrial projects. For example,
industrial projects often feature unusual baselines and calculation methods that require
unconventional approaches to data collection.
� Using updated simulation software to estimate above code savings. New construction and
major renovations of commercial buildings are covered under Title 24, using compliance
software approved by the CPUC. Projects permitted under Title 24, versions 2013 and
newer, use EnergyPlus as the simulation engine, which requires different skills and tools.
� Obtaining sufficient project data to validate calculation tool inputs. This includes building
and project plans, equipment submittals in electronic format, and supporting spreadsheets
with “live” calculation cells.
5.2.1.3 Site-Specific NMEC Projects
Our evaluation planning will also address the unique aspects of NMEC projects within the site-
specific M&V planning explained above. To meet the evaluation goal of minimizing ex post
evaluation risk, we will confirm the availability of data at any point in the entire project life-
cycle. If possible, it will also be advantageous to have data to support parallel analyses to
triangulate estimates, using both top-down and bottom-up approaches.
The key to having consistent and sufficient data is to set up data acquisition requirements,
relationships, and processes as early as possible. The evaluation analyst will gather key data for
the relevant time periods (baseline or annual reporting period), such as:
1. Electric and/or gas usage (15-minute, hourly, or daily intervals) with corresponding
measurement boundaries,
2. Weather conditions,
3. Occupancy,
4. Ex ante savings,
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 23
5. Project-related activities and dates of activities, and
6. Non-routine events, that is, significant non-project related changes, such as tenant
improvements or other capital projects, setpoint or scheduling changes, occupancy changes,
or utility infrastructure failures.
Occupancy data can be particularly important for certain building types, such as large office
buildings, hotels, and hospitals. If occupancy data is planned for at the start, then it can be
included as an independent variable in the model. If not planned for, there is an increased
likelihood that any changes in occupancy would need to be treated as a non-routine event. For
building types that have frequent or ongoing changes in occupancy, such as hotels, it is
unacceptable to treat occupancy changes as non-routine.
For each sampled NMEC project, we will review the model from the project and create an
NMEC model using the ECAM™ tool. To prepare for analysis, we will clean all meter data
using statistical methods, and, if any data issues arise, we will confirm with onsite visual
verification. We will cross-check historical weather data using one or two additional weather
stations. We will develop a model for the reporting period based on meter and weather data. We
will calculate model fitness statistics on the reporting period model and compare them to the
criteria specified in the draft Guidance for Program Level M&V Plans: Normalized Metered
Energy Consumption Savings Estimation in Commercial Buildings17. If the reporting period
model does not meet desired model fitness, we will mention that information in our report. If
the reporting period model is acceptable, we will then apply typical weather to establish typical
reporting period consumption. If we identify any issues with any of the models in our analysis,
we will reconcile their results.
We will then estimate the normalized savings for the project by taking the difference between
baseline typical consumption and reporting period typical consumption. If significant non-
routine events occurred during the baseline or reporting phases, we will estimate the impact of
those events and adjust the savings for the project. If we find significant differences after
comparing our savings estimate to the program savings estimate for the project, then we will
investigate the reasons using a thorough process developed for the early M&V study of the
PG&E Commercial Whole Building Demonstration.18 We will report the magnitude of the
difference in savings estimates and provide explanations for those differences.
5.2.2 Determine Ex Post Conditions
As outlined in each site-specific M&V plan, we will determine ex post conditions, and, in some
instances, reconfirm baselines by conducting site visits and interviews with facility personnel
who are familiar with the sampled projects and site operations. Key elements of these efforts are
described below.
17 Lawrence Berkeley National Laboratory. 2018. Guidance for Program Level M&V Plans: Normalized Metered Energy
Consumption Savings Estimation in Commercial Buildings Version 1.0. San Francisco, CA: CPUC. 18 SBW. 2019. Commercial Whole Building Demonstration Early M&V Report. San Francisco, CA: PG&E..
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 24
Coordinating with the customer
As described above, we will contact the selected project sites as part of the M&V planning
process. After first attempting to reduce disruption to facility personnel by determining if any
projects share contacts or sites, our field engineers will coordinate with program implementers
to contact customers. Depending on the requirements in the site M&V plans, we will cluster
interviews and site visits by location to schedule them at times that create the least burden on
customer staff and at times during which data can be appropriately gathered, even if this means
that the interview or visit occurs outside normal working hours, or if we must wait until the
facility has downtime.
Similarly, to the extent possible we will maximize customer convenience and data security by
enabling them to fulfill data requests through secure electronic file exchanges. If we need to
obtain any other required information, we will continue our discussions with the customer
representative or members of the customer’s engineering or operations staff. Likewise, if we
need information from one of the vendors associated with the project, we will ask the customer
for the best way to contact that vendor, and then we will follow up with that contact.
At the outset of site visits, we will determine all safety and access requirements and make
necessary arrangements to conform to the customer’s requirements. In all cases, we will comply
with general OSHA safety regulations and NFPA70e requirements for arc flash protection when
supervising electrical power measurements.
Visiting project sites
We will visit customer sites to inspect the systems and equipment affected by their projects. The
primary objective of the inspection is to determine whether their project was completed and if it
is producing savings. During our site visit we will confirm the make and model of the project
equipment and determine whether its installation and operation conform to the project’s intent.
We will confirm current and anticipated equipment operation by interviewing the customer’s
staff and by inspecting equipment control settings. We will record our conversations and take
pictures to document the visit and facilitate quality control review by other members of our
team.
Based on the findings of our site visit and on our interviews, phone calls, and email exchanges
with customers and vendor staff, we will identify any appropriate corrections to savings and
cost-effectiveness parameters, and then we will document those differences in a manner similar
to that used during the file review.
While at the customer facility, we will collect data in accordance with the customized M&V
plan, adjusting our efforts as necessary based upon the operations found onsite and on actual
data. For example, we have found that sometimes our onsite customer contacts are uncertain of
the data trends in their systems, so it may be necessary to adjust our M&V plans based on actual
conditions found at the site. Our team is experienced at extracting data from a variety of
trending systems, and we frequently work with onsite personnel and third-party suppliers to
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 25
obtain trend data. However, if trend data is not available, we may have appropriate data logging
equipment installed by electricians after we receive approval from the customer.
Our team is well versed in the practice of data logging. We are familiar with the use of current-
only loggers, true-RMS power data loggers, light sensors, temperature and humidity loggers,
and various spot measurement equipment. Our team will rely on our best professional
judgement to determine appropriate conditions for each type of measurement and which
locations are best to deploy the equipment.
5.2.3 Re-Estimate Project Gross Savings
Once we have obtained the appropriate equipment readings or trend data for a site, we will
proceed to analyze the savings. For each sampled project we will re-estimate gross savings by
making one or more corrections to the inputs or algorithms of the ex ante model(s) that were
used by the PAs to estimate the original savings claim. These corrections will be based on the
findings from our determination of ex post conditions. The corrections will fall into the
following four categories.
1. Correct Algorithm Errors. These corrections could apply to custom spreadsheet models
where we find formula errors. It is beyond the scope of this effort to correct algorithmic
errors in building simulation programs or other specialized calculators such as AIRMaster or
MotorMaster.
2. Revise Algorithms. These revisions could apply to any type of model used to estimate
project savings. Our intent with these revisions is to achieve a model that more reliably
estimates the impact of the project on affected systems and equipment.
3. Modify Baseline Characteristics. These modifications could apply to any type of project,
ranging from minor changes to the capacity, efficiency, operating hours, or control
sequences of the affected equipment to major changes due to the imposition of code or
standard practice features. In cases where a project was misclassified as replace on burnout
and we need to model it as early retirement, this would also require us to establish the
characteristics of the second baseline, which would determine savings after the remaining
use life of the initial baseline conditions, i.e., at the future point in time where normal
replacement would occur. In addition, we would need to establish the justification for early
retirement by collecting customer statements that the existing equipment would have been
operational for the RUL period.
4. Modify Measure Characteristics. As with the baseline characteristics, these modifications
could apply to any type of project. They would reflect the capacity, efficiency, operating
hours, or control sequences verified via direct observation or through our contacts with
customer staff or vendors.
Each of these types of corrections and modifications will be applied in the sequence listed
above. Where appropriate, we will calibrate modelled energy use to post-retrofit billing records.
In cases where short-term metering occurred, that metering data will be incorporated into the
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 26
analysis. For sites where measure savings are relatively large compared to billed energy use, we
may estimate savings using billing regression techniques or we may compare pre and post
installation billing records with the savings estimates that were calculated by algorithm. We will
review savings at both the measure and project levels, and we will document any differences
between our evaluation savings estimates and original program savings estimates.
Typically, this analysis will be performed in Excel, although additional tools such as eQUEST
or R may be used if the complexity or volume of data requires more specialized analysis. For
projects that exhibit weather dependence, we will obtain local weather data and perform
regression analyses for baseline and efficient conditions, each of which will be normalized to
TMY3 data. For some industrial projects, such as compressed air, the use of operational
equipment may vary with production levels. In these cases, we will request long term
production or operations data and normalize both baseline and efficient energy use to typical
operations throughout a year.
In general, our analysis will be based on conditions as they were observed, and our analysis will
be applied to the first year after measures became operational. If we find equipment that is
temporarily out of service for maintenance or some other reason, we may apply adjustments to
first-year extrapolations. We also reserve the right to assess the most appropriate long-term
conditions on which to base measure lifetime savings. Our data collection process will identify
when and why operations changed, and we will document any changes that are expected in the
future. This will enable us to base our savings analysis on an average long-term condition or on
a condition that is expected to remain stable over time.
In some cases, particularly for large industrial gas processing facilities, project savings may
represent a significant portion of overall billed usage on a meter. In those cases, we may request
the site’s billing data, so we can determine savings in accordance with IPMVP Option C. While
interval data is preferred for billing-based analyses, we recognize that it is not always available,
particularly for gas projects. If metered billing data is weather sensitive, we will weather
normalize it. Similarly, we will normalize billed usage to production to adjust for any variations
in operations. While billing analysis is not the most common method for determining savings at
industrial sites, it can be appropriate for large gas projects, particularly in unheated facilities.
Similarly, in cases where there are multiple electric meters at a site and the project has affected
the load on one meter significantly, such as at some chiller plants, normalized billed usage can
be the most accurate reflection of savings.
After each correction type is applied, we will record the new gross savings estimates and
associate these estimates with each correction type in our evaluation database. For each
evaluated project, we will summarize our findings from the interviews and site visits in a
project-specific M&V report. These site reports will be “sanitized,” in that they will not contain
information that could identify the customer, except for PA-assigned and PA-trackable
identification numbers.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 27
5.3 Net Savings Evaluation
Our approach to estimating net savings is to estimate a net-to-gross ratio (NTGR) that will be
multiplied by gross savings to obtain net savings. Our estimation of the NTGR to support the
estimation of net savings relies on these documents:
� TechMarket Works. 2006. California Energy Efficiency Evaluation Protocols: Technical,
Methodological and Reporting Requirements for Evaluation Professionals {a.k.a. Evaluators’
Protocols}. San Francisco, CA: CPUC.
� Ridge, Richard, Ken Keating, Lori Megdal, and Nick Hall. 2007. Guidelines for Estimating
Net-To-Gross Ratios Using the Self Report Approach. Prepared for the California Public Utilities
Commission.
� CPUC. 2015. Methodological Framework for Using the Self- Report Approach to Estimating Net-to-
Gross Ratios for Nonresidential Customers. San Francisco, CA: CPUC.
� Itron. 2015. 2013 Custom Impact Evaluation Industrial, Agricultural, and Large Commercial. San
Francisco, CA: CPUC.
� Itron. 2017. 2015 Custom Impact Evaluation Industrial, Agricultural, and Large Commercial. San
Francisco, CA: CPUC.
� TechMarket Works. 2006. The California Evaluation Framework. San Francisco, CA: CPUC.
� Ridge, Richard, Phillipus Willems, Jennifer Fagan and Katherine Randazzo. The Origins of
the Misunderstood and Occasionally Maligned Self-Report Approach to Estimating the
Net-To-Gross Ratio. Presented at the International Energy Program Evaluation Conference
in August 2009.
5.3.1 Questionnaire Development
To ensure consistency between this evaluation effort and former custom evaluations, we will
base our development of the new 2018 and 2019 NTGR survey instruments on the
questionnaires designed for the evaluation of the 2015 and 2017 Custom programs. As we
review those former instruments we will draw upon pertinent lessons learned from those
evaluations, as well as drawing upon other recently completed evaluations and methodological
reviews by PWP, Evergreen Economics, Tetra Tech, NMR Group, and DNV-GL. We will also
coordinate closely with DNV-GL and the rest of the Group A evaluation team to ensure
consistency between our current evaluation efforts.
In all, we will develop three questionnaires, one for each level of rigor: basic, standard, and
enhanced.
To assess the participants who were assigned to the basic and standard levels of rigor, we will
use a computer-assisted telephone interview (CATI) survey, as was done for the evaluation of
the 2015 Custom program. To further improve consistency with the Group A evaluation effort,
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 28
we will field the CATI surveys using the same firm that they are working with. The basic survey
will take approximately 20 minutes to complete, while the standard survey will take
approximately 30 minutes.
For participants assigned to the enhanced level of rigor, personal telephone interviews will be
conducted by our staff of highly-trained professionals whose knowledge and experience are
commensurate with the interview requirements. Due to the complex nature of these projects,
our interviewers will also be assisted by an experienced engineer. Potentially, these interviews
will involve conversations with several people who were involved in the project, including the
primary decision-maker, operations staff, vendor representatives, program staff, and other
decision influencers.
Once the draft versions of each questionnaire are ready, we will pre-test each of them on two
participants apiece to ensure that the survey length is not onerous, the wording is clear, the
question sequence is appropriate, and skip patterns and consistency checks are functioning
correctly.
5.3.2 Data Collection
To achieve the highest possible response rate, we will use several strategies. First, the team
responsible for estimating gross savings will work with site personnel (referred to as operations
staff) to determine whether they or someone else was the key decision maker. If they are the
key decision maker, the team will inform them that they will be contacted by a representative of
Opinion Dynamics to schedule the NTGR interview. They will also underscore the importance
of participating in this interview and that their answers will be kept strictly confidential.
If they are not the key decision maker, the team will proceed to conduct a brief interview with
the operations staff to collect additional information that will be used in the analysis of program
influence as well as to obtain the name and contact information (e-mail address, US Postal
address and telephone number) of the key decision-maker who will then be contacted by
Opinion Dynamics to schedule the NTGR interview.
Second, we will send each sampled participant, identified as the key decision maker, an e-mail
that: 1) explains that the CPUC is sponsoring this evaluation of the Custom program, 2)
underscores the importance of this evaluation and the confidentiality of their responses, and 3)
provides the names and telephone numbers of IOU program managers and key CPUC staff who
can verify the legitimacy of the survey. The e-mail will also include a list of the topics and
questions covered in the survey so that the customer can prepare in advance. In cases where an
e-mail address is not available, we will send a letter containing the same information via US
mail.
We anticipate that we will not obtain completed NTGR surveys for all 235 participants in the
gross savings sample. The response rate for the 2015 Custom program NTGR survey was 75%.
So, to complete a total of 235 NTGR surveys and to complete the additional 65 sites needed to
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 29
reach our goal of 300 completed NTGR surveys, we will increase the total number of sampled
sites by approximately 100.
When the surveys are complete, we will provide the CPUC with a detailed sample disposition
table that reports response rates in a manner consistent with the standards developed by the
American Association for Public Opinion Research. We will report other metrics and survey
outcomes as requested by Energy Division staff.
5.3.3 NTGR Analysis
Once data has been collected and cleaned, we will estimate the life-cycle-savings-weighted
NTGR for each domain using the approach described in section 6.2. We will then conduct
analyses of NTGR reliability, NTGR sensitivity, and NTGR drivers.
5.3.3.1 NTGR Reliability Analysis
We will use Cronbach’s alpha19 to assess and report the reliability of our estimated NTGRs to
ensure that our measure of program influence is internally consistent. Cronbach’s alpha is a
widely accepted measurement of the internal consistency and reliability of a multivariate
measurement that can be used to measure an underlying construct such as program influence.
Alpha represents the reliability of an overall set of measurements such as program attribution
index scores. Generally, alpha should be 0.70 or above. While Cronbach’s alpha increases as the
number of items in the scale increases, we usually have only three to four items distilled from
multiple questions. So even with consistency checks, inconsistencies invariably emerge, leading
to low inter-item correlations. As a result, an alpha lower than the normal threshold of 0.70
might be acceptable. For large energy savers, we will consider additional quantitative and
qualitative data that is not reflected in the alpha. When at least two raters are involved in
independently estimating the NTGR, we will also calculate inter-rater reliability.
5.3.3.2 NTGR Sensitivity Analysis
To assess the stability and possible bias of the estimated NTGR, we will conduct sensitivity
analyses, which may involve changing weights and changing the questions used in estimating
the NTGR.
5.3.3.3 Assessing the Drivers of the NTGR
Behind the net-to-gross ratios calculated for each project are several contextual factors that
might have influenced the project, either directly or indirectly. To assess the drivers of the
NTGRs, we will employ a two-pronged approach that combines a quantitative regression
analysis with a qualitative examination of the factors that influenced customer decision-makers.
19 Cronbach L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 30
Together, these analyses will enable us to make actionable recommendations to improve
program influence.
Quantitative Approach
We will use a regression analysis to examine the program, non-program and participant
variables that might be associated with high free-ridership (low NTGR) or low free-ridership
(high NTGR). The intent of this analysis will be to look beyond the numerical responses used in
the NTGR algorithm to better understand the factors that influenced the decision-makers on the
project. The final file will include analysis of variables such as average monthly electricity bill,
importance of the rebate, importance of the technical assistance, previous program experience,
square footage, presence of a company policy regarding the purchase of energy-efficient
equipment, vendor influence, timing of the program’s involvement and at what point in their
decision-making process the customer became aware of the program, the number measures
installed, the end use affected by the new equipment, and , and the level of NTGR rigor they
received.
We will specify a variety of regression models, and, for each model, regression diagnostics
including influential observations and collinearity will be reported followed by any will undergo
any necessary adjustments. We will also calculate a variety of weights for use in the analyses.
For example, the effect of a corporate energy policy that is rigorously enforced might be
negative while the effect of the rebate on reducing the simple payback period might be positive.
The general form of the models is this equation:
����� = � + ∑ � + ��� ��� + �� where:
����� = NTGR for the ith customer
� = the intercept
��� = a vector of customer and program/non-program characteristics for the ith customer
� = a vector of k coefficients that reflect the change in the NTGR associated with a one-unit change in the kth explanatory variable
�� = captures the differences in NTGRs among the various customers that are not explained by the model
This regression analysis will lead to an improved understanding of the factors underlying the
survey results. It will also help us to better understand program influence by highlighting the
characteristics of the strongest and weakest groups of projects. These insights will in turn help us
to prepare a set of actionable recommendations to improve program influence.
Qualitative Approach
As with the quantitative analysis, the goal of qualitative analysis will be to highlight the factors
(both program and non-program) and characteristics of the groups of projects with both the
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 31
strongest and weakest levels of program influence. As such, we will examine the same set of key
contextual factors that we considered in the regression analysis. This qualitative analysis will be
performed for the lowest and highest NTGR ratio quartiles, i.e., the group with the lowest
NTGRs and the group with the highest NTGRs. As with the quantitative analysis, the insights
we gain from this effort will help us to prepare actionable recommendations to reduce free
ridership.
5.4 Cost, Lifetime, and Load Shape
This section describes our approach to evaluating the measure costs, lifetimes, and load shapes
that we will use to estimate cost-effectiveness, as described in section 6.3.
� Incremental measure cost. We will evaluate measure costs by reviewing project
documentation to verify that valid and appropriate costs were included per the rules defined
in the 2018 Statewide Customized Offering Procedures Manual for Business20. For add-on
measures, we will consider full measure cost. For early-retirement measures, we will confirm
that the early-retirement cost was correctly calculated as the total installation cost minus the
net present value of the total cost that would have been incurred to install an ISP measure at
the end of the remaining useful life period. For replace on burnout measures and new
construction measures, we will verify the marginal cost of implementing the measure by
comparing the incremental measure cost to the measure cost of equipment that meets code
or industry standards.
� EUL and RUL. We will use all the available information for each claim to assign it the most
appropriate EUL from DEER. If we determine that the claim is an early-retirement
measure, we will assign it a RUL equal to one-third of the EUL, unless the PA has
substantiated an alternate RUL.
� Load shape. For each of the claims comprising the sampled project, we will select the most
appropriate load shape from the load shape library in the CPUC’s cost effectiveness tool.
20 PG&E, SCE, SoCalGas, SDG&E. 2018. 2018 Statewide Customized Offering Procedures Manual for Business. San
Francisco, CA: CPUC.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 32
6 Portfolio Savings and Cost-Effectiveness
We will estimate gross savings, net savings, and cost-effectiveness for the portfolio of custom
projects claimed in 2018 and 2019. We will prepare separate estimates for each of the domains
in the sample design and for all projects associated with each PA.
6.1 Gross Savings
We will calculate the domain-level ex post gross savings and the achieved precision for each fuel
type using the stratified mean estimation method and the stratified ratio estimation method21.
For the stratified mean estimation method, the steps are as follows:
1. Calculate the mean ex post gross savings
�̄st=����̄��
h=1
where:
Wh = N
N
h = which is the stratum weight
Nh = population of stratum h
N = population of domain
y h = the mean of y for stratum h
y st = the mean resulting from a stratified random sample (st for stratum).
2. Calculate the domain ex post gross savings
� = ����� The realization rate is the ratio of the domain ex post gross savings to the domain ex ante
gross savings:
�� = YX
where:
X = the sum of the ex ante savings in the domain
3. Calculate the variance of the mean, S2
21 Cochran, William G. (1977). Sampling techniques. New York: John Wiley & Sons.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 33
∑∑=
L
1=h
2
hhL
1h h
2
h
2
hst
2 sW -
n
s W = )y(
NS
where:
2
hs = the stratum variance
nh = the stratum sample size
The second term in the equation represents the finite population correction.
The achieved relative precision at 90% confidence is:
�" = 1.645'(����)����
For the stratified ratio estimation method, the steps are as follows:
1. Calculate the ratio (realization rate), b
∑
∑
=
==n
i
ii
n
i
ii
xw
yw
b
1
1
where:
wi = Nh/nh
yi = ex post savings
xi = ex ante savings for sample point i
Large energy savers, selected with certainty, will be assigned a sample weight of 1. Other
projects, selected at random, represent more than one project in the population and thus will
be assigned a weight greater than 1.22
2. Calculate the residual of each sampled point, ei = yi−bxi, then the standard error across
strata for each domain is:
( )( )
∑
∑
=
=
−
=n
i
ii
n
i
iii
xw
eww
bse
1
1
21
22 Savings for projects in the exclusion stratum will be passed through, i.e., y7i=xi.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 34
and the achieved relative precision at 90% confidence is:
( )b
bserp
645.1=
We will also apply the steps above to derive PA-level ex post estimates for electric and gas
savings. We will determine which method, stratified mean estimation or stratified ratio
estimation, gives the most reliable estimate of precision for the PA-level realization rate.
In addition to reporting the domain ex post gross savings and achieved precision, we will also
provide a table of the sample disposition that will include the response rate and the number of
replacements used in each domain. We will report any potential for non-response bias, and, if
necessary, weight the results to correct for any bias.
6.2 Net Savings
Within each of the sample domains that we established for estimating the gross savings, we will
estimate the ex post net savings for the population in each domain by multiplying the ex post
gross savings for the population by the sample-based ex post life-cycle savings-weighted NTGR.
The resulting ex post net savings will then be summed up across the domains related to each PA
to calculate the total ex post net savings along with the achieved relative precision. We will then
sum the PA ex post net life-cycle savings across the five domains to arrive at the estimate for
total statewide portfolio savings. First, for the two parameters, the ex post gross savings (GS)
and the NTGR, the 90% relative precision (RP) is calculated. Next, the 90% relative precision of
the net savings for each domain (D) is calculated in a way that, using the equation below,
accounts for the propagation of errors involved in multiplying these two parameters.23
2
D
2 )rp(NTGR )(SavingsNet Domain += DGSrpRP
where:
)( DGSrp = the 90% relative precision of the ex post gross savings
)( DNTGRrp = the 90% relative precision of the ex post NTGR
The error bound of the domain net savings will then be calculated using this equation:
SavingsNet DomainRPNSEB DD ×=
23 TecMarket Works, Megdal & Associates, Architectural Energy Corporation, RLW Analytics, Resource Insight, B &
B Resources, Ken Keating and Associates, Ed Vine and Associates, American Council for an Energy Efficient Economy, Ralph Prahl and Associates, and Innovologie. (2004). The California Evaluation Framework. Prepared for the
California Public Utilities Commission and the Project Advisory Group; Taylor, John R. (1997). An Introduction to
Error Analysis: The Studies of Uncertainties in Physical Measurements. Sausalito, CA: University Science Books.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 35
where:
DEB = the 90% error bound of net savings for a domain
DNS = the net ex post savings for a domain
Once the error bound of net savings for each of the domains is calculated, the error bound for
each PA and the statewide portfolio can be calculated using this equation:
) Domain(EB .....) Domain(EB ) DomainEB( 2
n
2
2
2
1 ++=PortfolioEB
The equation for EBD above is based on these assumptions:
1. There are no interactions between the domains,
2. Each of the individual domains has been evaluated independently, and
3. Each evaluation has provided an unbiased estimate of the actual savings of the
corresponding domain
The result is a simple consequence of (a) the fact that the standard deviation of a sum of
statistically independent random variables (e.g., the estimated savings of each program) is the
square root of the sum of the squares of the standard deviations of each of the random variables,
and (b) the error bound being defined as 1.645 times the standard deviation.
Finally, the relative precision of the portfolio for each PA and the statewide portfolio, defined as
a combination of two or more domains, will be calculated using this equation:
Portfolio
Portfolio
PortfolioNS
EBRP =
where:
PortfolioEB = the error band of the PA portfolio or statewide portfolio.
(Only the net savings estimated via ex post EM&V are included in
PortfolioEB . It excludes net savings from measures whose parameters
were not evaluated, were passed through, or were DEER based. It also excludes interaction effects.)
PortfolioNS = the net savings impacts (kWh, kW or therm) of the PA portfolio or
statewide portfolio. ( PortfolioNS includes all net savings. That is, the
net savings estimated via ex post EM&V, as well as the net savings from measures whose parameters were not evaluated, were passed through, or were DEER based. It also includes interaction effects.)
The expected relative precision will improve anytime two or more domains are combined for
reporting purposes due to the increase in the number of observations. The best level of relative
precision will be for the portfolio that includes all the domains. Results will also be reported by
electric and gas separately for the same combinations.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 36
6.3 Cost-Effectiveness
We will use the CPUC cost-effectiveness tool (CET) to compute life-cycle costs and benefits for
each claim, using our ex post estimates of net savings by baseline, EUL/RUL, load shape, and
incremental costs. These cost and benefits will be aggregated to the domain level and PA level
using methods described above and will then be used to compute cost-effectiveness ratios such
as total resource cost (TRC).
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 37
7 Data Management and Quality Control
We will develop an Excel workbook to standardize how we document all primary data
collection and modelling for each project and for the constituent claims. The workbook will
have a series of locked tabs that contain data-entry ranges and that validate each value entered.
Instructions for entering data will appear in adjacent cells. Our analysts will add additional tabs
to the workbook to document the models and inputs used to calculate savings.
Our senior engineering staff will:
� Develop the workbook
� Train our project-review staff
� Provide quality control
� Resolve problems
In their quality-control role, our senior engineering staff will review each of the complete
project-evaluation workbooks and interact as necessary with our analysts to ensure that all
projects are treated consistently.
Our data-processing team will compile data for the project workbooks and associate that data
with claim data. They will apply additional quality-control tests to the data and refer any issues
that need resolution to the assigned analyst.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 38
8 Task Plan
This section describes how we will complete the ex post evaluation of 2018 and 2019 custom
projects. Its structure corresponds to the budget and schedule portions of the workplan.
8.1 Workplans and Updates
This document is our initial IALC workplan. We will submit an updated IALC workplan in
early 2020 to refine and improve our approach, and to respond to changes as programs evolve.
8.2 Data Collection and Sampling Plans
8.2.1 Sample Frame
We will obtain data from CEDARS on all PA claims for 2018 and collaborate with the Group
A evaluation team to determine how the custom claims will be segregated from all other claims.
This will be done as soon as the PAs confirm the 2018 claim has been finalized. Similarly, we
will obtain and identify custom claims for each quarter of 2019.
8.2.2 Review of Reported Custom Claims
We will review the 2018 claims to identify whether additional information is required to support
a review of compliance with CPUC policies. We will issue requests for any necessary additional
information that is identified during this review. Once we have all the requested data, we will
complete the review as described in section 3.2. We will repeat this process for the 2019 claim.
8.2.3 Sample Selection
We will apply the 2018 sample design to the sample frame and select the ex post evaluation
sample, including enough projects to satisfy the sample-size objectives for gross- and net-savings
evaluations. We will apply a similar design to select samples representing each quarter of 2019,
with the final quarter’s sample selected after the PAs confirm that the 2019 claim is final.
8.2.4 Data Requests Related to Sampled Custom Projects
We will request that PAs upload all files for the 2018 sampled projects. We will review these
files and then request that PAs provide any missing critical data, models, or documentation. We
will repeat this process for each of the 2019 quarterly project samples.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 39
8.2.5 Site Data Collection
For each project in the 2018 and 2019 gross savings samples, we will develop an M&V plan that
will serve as a roadmap to guide the customer interviews, site visits, data collection, and the
procedures to estimate savings. Each M&V plan will be customized to the site conditions, the
type of measures installed, and the assigned rigor level for gross savings and NTGR estimation.
We will collect all the data specified by each M&V plan, including the data obtained by
completing the NTGR survey. We will also expand the number of projects in the sample for
each of the 2018 and 2019 NTGR surveys to ensure that we obtain a sufficient number of
completed surveys for the analysis.
8.3 Gross Savings Estimates
For each project in the 2018 and 2019 gross sample we will process and prepare an analysis-
ready version of the data collected. We will finalize the M&V procedures to reflect the observed
conditions for each project. We will then estimate gross savings for each claim comprising each
of these projects. For claims involving normal replacement measures, we will estimate savings
for the first baseline period. For early-retirement measures, we will estimate savings for both the
first and second baselines. We will use the results from the sample to estimate gross life-cycle
savings for each sample domain and for each PA.
In addition, for each of the claims comprising the sampled projects, we will estimate
incremental measure costs and assign the appropriate measure lifetime and load shapes.
8.4 Net Savings Estimates
We will pre-test the NTGR questionnaires and make any necessary revisions. We will then
complete the NTGR with 2018 and 2019 net savings samples. We will process the data from
each survey and prepare an analysis-ready version of the data, which will then be used to
estimate NTGR for each of the claims comprising these projects. We will use the results from
the sample to estimate net life-cycle savings for each sample domain and for each PA.
Using the NTGR estimates for each of the claims, we will estimate life-cycle costs and benefits,
and then use the sample to estimate cost-effectiveness results for each sample domain and PA.
8.5 Final Reports
We will develop a draft and final evaluation report for 2018 and 2019. These reports will
document our evaluation methodology and present our estimates of gross and net savings and
cost-effectiveness. These reports will include an analysis of the reasons for corrections to the
reported claims, including separate estimates of the change in savings due to our review of the
reported claims (see section 3). The reports will also include actionable recommendations
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 40
designed to improve the PA’s savings claims for future custom projects and to improve future ex
post evaluation methods.
8.6 Data Documentation
We will document all our primary data collection, modelling, and data processing procedures so
that our results are transparent and can be reproduced by other parties.
Workplan for 2018-19 IALC Custom Impact Evaluation
SBW Consulting, Inc. 41
9 Budgets and Schedule
This section presents our budget and schedule for evaluating the 2018 and 2019 IALC
programs.
9.1 Evaluation Budget
The IALC evaluation is part of the larger budget for all early feedback and ex post evaluation
(Group D contract deliverables 9, 10, 13, and 17). The other portion of this larger budget is for
the evaluation of strategic energy management (SEM) projects. Table 8 presents our budget in
total and separately for SEM and for the IALC activities discussed in this workplan.
Table 8: IALC Evaluation Budget
Budget by category ($)
Deliverables IALC SEM Early feedback and ex post evaluation (Total
of IALC and SEM)
Workplans and updates 168,000 168,000 336,000
Data collection and sampling plans 129,000 43,000 172,000