Information Technology (IT) Volume 2 – Capitalized Software...SUMMARY • This Volume presents SCE’s request for $809.1 million in capitalized software expenditures for the 2016-2020

Application No.: A.16-09- Exhibit No.: SCE-04, Vol. 2 Witnesses: D. Bauder

D. Bernaudo T. Boucher J. Castleberry T. Condit G. Haddox T. Inlander P. Joseph

J. Kelly D. Kempf J. Lim S. Nagoshi D. Pierce M. Provenzano J.P. Shotwell J. Tran

(U 338-E)

2018 General Rate Case

Information Technology (IT) Volume 2 – Capitalized Software

Before the

Public Utilities Commission of the State of California

Rosemead, California

September 1, 2016

SUMMARY

• This Volume presents SCE’s request for $809.1 million in capitalized software expenditures for the

2016-2020 forecast period.1 These funds will support SCE’s core business processes, improve

customer transactions and outage notifications, and support the modernizing of the grid to improve

safety and reliability.

• This volume includes testimony on the following:

o An Operating System Software request for $82.6 million, which provides foundational software

that manages computer hardware and enables business applications to perform daily work

functions and to operate across multiple technology platforms, such as web and application

servers, storage, and personal computers.

o Cybersecurity & Compliance capitalized software project requests for $308.9 million that will

continue to protect critical SCE systems, the electric system, and customer information from

growing cyber threats, especially as access points to the system expand. This also includes

requests that will support compliance with current and future NERC CIP standards. Included in

this capitalized software forecast is $99.1 million for Grid Modernization Cybersecurity.

o Technology Consolidation & Optimization capitalized software project requests for $41.7

million that will help to optimize our existing technology portfolio to increase efficiencies and

reduce costs.

o SCE’s Operating Units have requested $376.0 million specific capitalized software projects to

support critical business operations. This request reflects SCE’s robust review, prioritization, and

selection, which allows us to put forth a technology portfolio that provides the most benefit to

the company and our customers.

1 Refer to WP SCE-04, Vol. 2 pp. 1-4.

Capitalized Software Expenditures 2016-2020 Forecast

CPUC-Jurisdictional Only (Nominal $Million)

SCE-04: Information Technology Volume 2 – Capitalized Software

Table Of Contents

Section Page Witness

-i-

I. INTRODUCTION .............................................................................................1 T. Inlander

A. Summary of SCE’s Capitalized Software Request ................................1

B. Compliance Requirements .....................................................................1

C. 2015 Authorized versus Recorded .........................................................2

D. Capitalized Software Estimation............................................................3

II. OPERATING SYSTEM SOFTWARE..............................................................6 M. Provenzano

A. Operating System Software ...................................................................6

1. Project Description .....................................................................6

2. Need for the Project ...................................................................7

a) Business requirements ...................................................8

3. Scope and Cost Forecast ..........................................................11

a) Recorded Expenditures ................................................11

b) Forecast Expenditures ..................................................12

III. CYBERSECURITY & IT COMPLIANCE .....................................................15 G. Haddox

A. Perimeter Defense ................................................................................19

1. Program Description ................................................................19

2. Need for Program .....................................................................19


B. Interior Defense ...................................................................................21


2. Need for Program .....................................................................21


C. Data Protection.....................................................................................23


Table Of Contents (Continued)


-ii-


2. Need for Program .....................................................................24


D. SCADA Cybersecurity .........................................................................25


2. Need for Program .....................................................................26


E. Common Cybersecurity Services for Generator Interconnections ...................................................................................28

1. Project Description ...................................................................28

2. Need for Project .......................................................................29


F. Grid Modernization – Cybersecurity ...................................................30


2. Need for Project .......................................................................32


a) Alternatives Considered ...............................................36

G. IT Support for NERC CIP Compliance ...............................................37


2. Need for Project .......................................................................38



IV. TECHNOLOGY CONSOLIDATION & OPTIMIZATION ...........................43 J. Kelly




-iii-

A. Data Warehouse Consolidation ............................................................43


2. Need for Project .......................................................................44



B. Lotus Notes Migration .........................................................................46


2. Need for Project .......................................................................47



C. Backup and Disaster Recovery Optimization ......................................49


2. Need for Project .......................................................................50



D. Information Technology Projects less than $3 Million ........................51

V. OPERATING UNIT SOFTWARE PROJECTS ..............................................52

A. Customer Service Software Projects ....................................................52 J. Lim

1. SCE.com Strategic Upgrade/Stabilization ...............................52

a) Project Description .......................................................52

b) Need for Project ...........................................................53

c) Scope ............................................................................53




-iv-

(1) Remaining Scope to be Completed in 2016..................................................................53

(2) Completed Scope .............................................54

d) Recorded and Forecast Expenditures ...........................55

2. Digital Customer Self-Service .................................................56



c) Scope ............................................................................57

(1) Device Support.................................................58

(2) Electronic Billing and Payment Transactions .....................................................58

(3) Security and Authentication .............................59

(4) Website Functionality ......................................60

d) Forecast Expenditures and Cost-Benefit Analysis........................................................................62

e) Alternatives Considered ...............................................63

3. Alerts & Notifications ..............................................................64



c) Completed Project Scope .............................................65

d) Remaining Project Scope .............................................66

e) Recorded and Forecast Expenditures ...........................68

4. Meter Data Management System (MDMS) Upgrade ..............68 D. Kempf





-v-


c) Scope and Cost Forecast ..............................................70

(1) Phase 1 .............................................................70

(2) Phase 2 .............................................................70

d) Recorded and Forecast Expenditures ...........................71

5. Customer Service Projects Less than $3M ..............................72 D. Kempf, ...................................................................................................... D. Bernaudo ...................................................................................................... S. Nagoshi

B. Transmission & Distribution Software Projects ..................................72

1. Work Management Solutions ..................................................72 P. Joseph

2. WM - Portfolio Management ...................................................75



c) Scope and Forecast ......................................................78

(1) Alternatives Considered ...................................80

3. Scope Cost Management Tool (SCMT) ..................................80





4. Work Management Dashboard ................................................84



(1) Benefits ............................................................86




-vi-



5. Transmission Telecommunications Work Order Lifecycle ..................................................................................88



(1) Benefits ............................................................89



6. Click Schedule Refresh Release 1 & 2 ....................................92



(1) Benefits ............................................................94



7. Vegetation Management ..........................................................95





8. Pole Loading Application Replacement Tool ..........................97 T. Boucher


b) Recorded Costs and Forecast .......................................99




-vii-

9. Design Manager (DM) Refresh ...............................................99


b) Recorded Cost and Forecast.......................................100

10. Graphic Design Tool (GDT) and Tract Deployment Refresh ...................................................................................101

a) Project Description .....................................................101

b) Need for Project .........................................................102

c) Scope and Cost Forecast ............................................103

(1) Alternatives Considered .................................103

11. Consolidated Mobile Solution (CMS) ...................................104





12. Field Tools Upgrade ..............................................................106



a) Scope and Cost Forecast ............................................107


13. Enhanced Business Resiliency for Energy Management System (EBR) ..................................................108 G. Haddox



(1) Benefits ..........................................................109




-viii-



14. Comprehensive Situational Awareness for Transmission (CSAT) Phase 1 ...............................................110 T. Boucher



(1) Benefits ..........................................................112



15. Centralized Remedial Action Scheme (CRAS) Project ....................................................................................115



(1) Analysis of additional complex RASs ..............................................................119

(2) Generation Queue Completion ......................122

(3) Benefits ..........................................................124



16. RGOOSE Project ...................................................................130



(1) Benefits: .........................................................133

b) Scope and Cost Forecast ............................................134




-ix-


17. Energy Management System (EMS) Refresh ........................136



(1) Benefits: .........................................................137



18. Outage Management System (OMS) Refresh ........................138



(1) Benefits ..........................................................139



19. Distribution Management System (DMS) Refresh ................141


b) Recorded Costs and Forecast .....................................143

20. Grid Interconnection Processing Tool (GIPT) .......................144



(1) Benefits ..........................................................147



21. Grid Analytics Applications (GAA) ......................................150




-x-



(1) Benefits ..........................................................152



22. Long-Term Planning Tools ....................................................155



(1) Benefits ..........................................................159



23. Grid Connectivity Model .......................................................161



(1) Benefits ..........................................................163



24. Transmission and Distribution Projects less than $3 Million....................................................................................167

C. Power Supply Software Projects ........................................................168

1. Generation Automation Upgrade & Control Systems Refresh .....................................................................168 T. Condit






-xi-

(1) Benefits ..........................................................169

c) Project Scope and Forecast ........................................169


2. Dam Monitoring and Surveillance .........................................170


b) Project Need ...............................................................171

(1) Benefits ..........................................................171



3. CAISO Market Enhancements Program (IMEP) ...................173 J. Tran



(1) Benefits ..........................................................174



4. Energy Planning Platform Upgrade (EPP) ............................176



(1) Benefits ..........................................................177



5. PCI Replacement ...................................................................178





-xii-


(1) Benefits ..........................................................179



6. Energy Trading and Risk Management (ETRM) System Replacement ..............................................................181



(1) Benefits ..........................................................182



7. Aggregated Demand Response (ADR) ..................................184



(1) Benefits ..........................................................186



8. Commodity Management Platform (CMP) ...........................187



(1) Benefits ..........................................................188






-xiii-

9. Generation Management System (GMS) Upgrade ................189


b) Recorded Costs and Forecast .....................................190

10. Power Supply Projects less than $3 Million ..........................191

D. Ethics and Compliance ......................................................................192 J.P. Shotwell

1. Enterprise Content Management ............................................192


b) Need for Project and Scope ........................................193

c) Cost Forecast ..............................................................198


2. Electronic Document Management / Records Management (eDMRM) .........................................................200





E. Finance Capital Projects ....................................................................203 D. Pierce

1. Plant Ledger Upgrade and Tax Module Installation ..............203





2. Corporate Projects less than $3 Million .................................207 J. Castleberry




-xiv-

F. Operational Services Capital Projects ................................................207

1. C-CURE 9000 ........................................................................207 D. Bauder





2. Operational Services Projects less than $3 Million ...............210 J. Castleberry

1

I. 1

INTRODUCTION 2

A. Summary of SCE’s Capitalized Software Request 3

SCE is requesting $809.1 million from 2016 – 2020 to implement needed capitalized software to 4

support the business capabilities of SCE Operating Units and enterprise-level systems for SCE.2 5

Table I-1 Capitalized Software Forecast3

(Nominal $Millions)

B. Compliance Requirements 6

In D.12-11-051, SCE was directed to “establish that proposed capital projects are necessary and 7

that SCE has prudently examined alternatives for cost-effectiveness before seeking Commission 8

approval.”4 SCE has considered alternatives to projects and has detailed those evaluations in this 9

testimony. 10

In D.15-11-021, the CPUC required that SCE “include its own forecast and the Commission’s 11

adopted forecast from the previous GRC alongside historical costs, and brief explanations detailing any 12

changes in the scope of a category.”5 The capitalized software projects requested address this 13

requirement within their respective testimonies. 14

2 This does not include the $208.8 million in capitalized software requested in SCE-04, Vol. 3 for the Customer

Service Re-Platform. 3 SCE expects to avoid capitalized software expenditures of $5.46 million in 2020 related to avoided

development for legacy software, as a result of the implementation of the Customer Service Re-Platform (CSR). Consequently, we do not include these costs in our capitalized software forecast. These cost avoidances are contingent upon approval of the CSR project costs as defined in SCE-04, Vol. 3. Should the Commission not adopt the proposed CS Re-Platform costs, the corresponding legacy software development expenditures should be added to SCE’s capitalized software forecast. Refer to WP SCE-04, Vol 3 pp. 169-171 for details on these costs avoidances.

4 D.12-11-051; Conclusion of Law #4. 5 D.15-11-021; pp. 224.

Testimony Volume Year 2016 2017 2018 2019 2020 2016-20 Forecast

SCE-04, Volume 2 Total 151.662 213.382 202.868 135.965 105.230 809.108

2

C. 2015 Authorized versus Recorded 1

Figure I-16 2015 Requested, Authorized and Recorded

(Constant 2015 $000)

As shown in Figure I-1, SCE’s overall capital expenditures were approximately $8 million below 2

the Commission’s 2015 authorized level. SCE manages our portfolio of capitalized software work in 3

consideration of available resources, competing business priorities, and technology deployment 4

strategies. In 2015, SCE invested more in enterprise and operating system software to make needed 5

upgrades to our server, storage, and desktop operating systems environments prior to vendor-published 6

end-of-life support for Microsoft, Cisco, and VMware Enterprise License Agreements, and in 7

preparation for our migration to Microsoft Office 365. Additionally, SCE underspent its authorized 8

levels for OU capitalized software. This was largely due to the deferral of the Customer Data Warehouse 9

6 Refer to WP SCE-04, Vol. 2 Bk A pp. 5-6.

($16,846)

$209 $8,751

$187,551 $185,286 $177,400

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,000

200,000

2015 Request 2015 Authorized CybersecurityVariance

Enterprise &Operating Software

Variance

OU CapitalizedSoftware Variance

2015 Recorded

Cybersecurity & Compliance Enterprise & Operating Software OU Capitalized Software

3

project from its originally planned date, as SCE found synergies in merging this investment with our 1

efforts to migrate to HANA Hadoop technology in the future, as discussed in Section IV, Technology 2

Consolidation & Optimization. 3

D. Capitalized Software Estimation 4

The SCE capitalized software projects in this testimony include a combination of both in-flight 5

and future initiatives. For in-flight projects, estimates are based on a detailed scope of work definition 6

and are supported by detailed labor, licensing, and infrastructure estimates captured through SCE’s 7

project estimation and management processes. Project estimates are refined during the project life cycle 8

when more project specific data is available, such as having completed Request for Information (RFI) 9

and Request for Proposals (RFP) from vendors and completing solution analysis resulting in a system 10

architecture recommendation. 11

Many Operating Unit initiatives to be implemented between 2018 and 2020 are in early stages of 12

the project life cycle and only initial business requirements and scope definition has been performed. For 13

these, the Operating Unit and IT leverage a high-level estimation methodology, which involves 14

analyzing historical similar Operating Unit capitalized software projects with equivalent scale and 15

complexity. Complexity is determined by the functionality of the application being purchased 16

(sometimes referred to as COTS – Commercial-Off-The-Shelf) or developed, and by the number and 17

complexity of integration points and data being transferred between systems. 18

The COTS and In-house Application Development forecast methodologies use spreadsheet-19

based calculations.7 The methodology includes assigning standard assumptions by project type that are 20

adjusted to account for known non-standard differences from the generic estimates of the estimation 21

model. These adjustments are based on IT and the Operating Unit’s current understanding of the 22

complexity of the initiative, the likelihood of leveraging an existing product, the integration points, the 23

data transferred amongst systems, the volume of data transferred, the data validation required through 24

the transfer, and the transfer intervals and automation required. SCE uses historical data from the last 25

five years of capitalized software projects to derive the generic estimation template parameters based on 26

the following six categories: 27

1. Simple Commercial-off-the-Shelf (COTS) Product 28

7 Refer to WP SCE-04, Vol. 1, Bk A pp. 101-112.

4

2. Medium Complexity COTS Product 1

3. High Complexity COTS Product 2

4. Simple Software Development 3

5. Medium Complexity Software Development 4

6. High Complexity Software Development 5

The project and technology complexity is determined by the functionality of the application 6

being purchased or developed as well as by the integration points and data being transferred between 7

systems. 8

The estimation methodology includes several type of labor resources identified below. 9

• SCE Labor - This is an estimate of the total team size and uses a blended billing rate 10

based on SCE resources (SCE employees and Managed Service Providers). The duration 11

of the high-complexity (large) development project is assumed to be 2,000 hours or 12

approximately one year. Medium complexity and simple (small) development projects 13

are assumed to require approximately six months and three months, respectively. The 14

actual duration may be more and the number of SCE employees could be less, meaning 15

the base cases are estimating a total effort and not necessarily a fixed timeframe. 16

• Vendor Labor - This is an estimate for the cost of labor provided by the software 17

company who created the COTS package. The software vendors provide us with 18

technical expertise of their software products and expertise from experience in deploying 19

their products at other companies. 20

• Systems Integrator (SI) - This is an estimate of the work performed by a person or 21

company that specializes in bringing together component subsystems into a whole and 22

ensuring those subsystems function effectively together. 23

• Business Consultants - Business Consultants assist internal Operating Unit resources 24

with many of the business activities. The Business Consultants can support development 25

of business requirements with stakeholders, assist with design reviews, and support 26

overall project management. Business Consultants are also often used if SCE is seeking 27

to deploy new business capabilities and would benefit from the experience and guidance 28

of industry experts. 29

• Technical Consultants - Technical Consultants assist when new technology is being 30

introduced or complex real-time interfaces require specific expertise. The technical issues 31

5

are often fewer than the complexity inherent in the business processes and are therefore 1

are not typically required to the same extent as the business consultants. 2

The estimation methodology also provides an ongoing annual O&M labor support cost based on 3

a percentage of the overall labor costs of the completed project. SCE uses historical ongoing support 4

costs of previously implemented capitalized software projects as a way to estimate the anticipated future 5

support costs. 6

This estimation methodology has been used to develop the cost forecasts for the projects 7

discussed in this volume. Details of the cost estimate for individual projects are provided in 8

corresponding workpapers.9

6

II. 1

OPERATING SYSTEM SOFTWARE 2

A. Operating System Software 3

Table II-2 Operating System Software8

Work Breakdown Structure (WBS)9 Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)

1. Project Description 4

SCE’s IT Operating Unit (IT) oversees an array of information technology infrastructure. 5

The computing systems composed of the computing hardware, such as servers in data centers and 6

laptops on end-user desks, and supporting software, are used by the SCE workforce and SCE customers. 7

Most of the supporting business software is Commercial-Off-The-Shelf (COTS) packages, which must 8

be updated on regularly with software patches and new releases. This section of testimony addresses the 9

core Operating System Software, which includes operating systems, business intelligence systems, 10

database management systems, cross-system integration tools, and the base platform of our Enterprise 11

Resource Planning system, SAP. This Operating System Software provides a stable and reliable 12

foundation for SCE’s critical business systems. 13

8 Refer to WP SCE-04, Vol. 2 Bk A pp 7-43. 9 Operating System Software: CIT-00-OP-CS-000030, CIT-00-DM-DM-000096, CIT-00-DM-DM-000064;

Database Platform Upgrade: CIT-00-OP-CS-000061, CIT-00-SD-PM-000246; Business Intelligence Tools Upgrade: CIT-00-ET-AE-000033; Enterprise Integration Tools Upgrade: CIT-00-ET-AE-000016; Enterprise Platform Core Refresh: CIT-00-CC-CC-000022.

ForecastOperating System Software 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 5Y TotalRecorded / Forecast 26.20 26.79 11.35 2.78 29.56 5.60 6.10 12.80 20.90 11.60 57.00

Database Platform Upgrade 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 3.00 - - 19.46 0.21 2.50 - - - 2.00 4.50

Business Intelligence Tools Upgrade 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 10.21 0.06 0.00 - - - 0.30 0.50 1.00 - 1.80

Enterprise Integration Tools Upgrade 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 21.39 3.11 3.11 2.46 0.17 - 0.30 1.00 1.00 0.30 2.60

Enterprise Platform Core Refresh 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 4.19 8.05 5.29 0.00 0.00 - 8.00 8.70 - - 16.70

OS Total 82.60

Recorded Forecast

7

2. Need for the Project 1

Operating System Software operates mainframe servers, midrange servers, and storage 2

devices in SCE’s data centers, customer call centers, and personal computers. This section presents the 3

business need to update Operating System Software, including any requisite testing and remediation 4

needed for applications to run on the updated operating system software. We have organized the 5

Operating System Software into five major areas: 6

• Windows 7, Windows Server 2012, AIX (Advanced Interactive executive product 7

from IBM), and Linux. This software is required to operate either our data center and 8

customer call center computer assets or end-user devices such as laptops and 9

desktops, and provide the foundation for the critical business applications operated by 10

SCE. Remote access software – such as Citrix and Virtual Private Networks (VPN) – 11

is also included. 12

• Database platforms are software products used to organize information so that it can 13

be easily accessed, managed, updated and analyzed. An example of a database 14

platform is the Oracle Database. The information derived from these databases 15

supports SCE’s critical business processes and capabilities, such as our asset 16

management, outage management, energy procurement, and financial management 17

applications and processes. It also supports SCE’s analytics platforms used to analyze 18

critical business information, such as Energy Price and Demand Forecasting, 19

Transformer Load Management, Voltage Management, and Storm Preparations. SCE 20

uses several database platforms, including Oracle, SQL Server, SAP Hana, and IBM 21

DB2. 22

• Business Intelligence (BI) is a set of methodologies, processes, architectures, 23

technologies, and services that help transform raw data into meaningful and useful 24

information for actionable business decision making. BI tools provide SCE the 25

essential capability to gain insight into many data to better translate that data into 26

actionable information and identify new opportunities and define and implement 27

strategies and services for cost, efficiency, process and operational improvements. 28

8

• SCE uses Enterprise Integration Tools to integrate and share data stored in various 1

applications, analytics systems, and between SCE and external third-party systems.10 2

These integration tools can also consolidate data into a data warehouse to provide a 3

unified view of data from different systems, which can allow for more complex data 4

mining and analytics to be completed to inform SCE business decisions. A few 5

examples of these tools include: 6

a. Application to Application integration; 7

b. IBM WebSphere ESB (Enterprise Service Bus) and SAP PI/PO ETL Tools 8

(Process Integration/Process Orchestration for Extract, Transform and Load); and 9

c. IBM Data Stage, SAP Data Services, etc. 10

• The Enterprise Platform Core Refresh will update our enterprise resource platform, 11

SAP, to a newer version supported by the vendor. SAP is the enterprise-wide system 12

used for our back-office financial management, human resources data, supply chain 13

management, work management, enterprise asset management, customer relationship 14

management, and governance, risk and compliance functions. The SAP system and 15

associated processes are essential for the operations of all the organizations within the 16

company. This refresh is not only critical for the ongoing operations of these business 17

processes, but it will also enable SCE to leverage its transition to SAP’s HANA 18

environment, which will increase performance while optimizing servers and storage 19

requirements. This SAP core refresh is also a key enabler to SCE’s gradual shift to 20

leverage cloud-based services maintained by vendors in the areas of Human Capital 21

Management and Supply Chain Management (Ariba), and the implementation of 22

additional customer relationship and customer care capabilities as they mature. 23

a) Business requirements 24

Operating System Software continues to be indispensable to SCE’s daily 25

operations. It allows SCE to operate effectively in highly complex and geographically diverse 26

computing environments. Software becomes obsolete during its lifecycle and requires updating on a 27

10 Examples of external 3rd party systems include CAISO (California Independent System Operator), Weather

Forecast Vendors, and JP Morgan Chase.

9

recurring basis as specified by the software manufacturers.11 A refresh of our Operating System 1

Software must manage and operate computing resources, while mitigating the ongoing potential risk to 2

critical business functions due to technology obsolescence and vendor obsolescence. 3

• Technology obsolescence results when the underlying technology that an 4

application uses is identified by its vendor to be phased out (no longer 5

approved for use). The reasons for technology obsolescence include, but are 6

not limited to, exposure to security vulnerabilities, unstable technology 7

platforms, and operating system changes. 8

• Vendor obsolescence (also known as Vendor Specified End of Life for 9

products) results when a COTS application or version of an application is no 10

longer supported by the vendor. 11

In the 2016-2020 period SCE will update the existing Windows 7, Windows 8, 12

and Windows 8.1 desktop operating systems to the current Microsoft platform, Windows 10. SCE has 13

approximately 17,500 end-point devices12 supporting operations across the company. The update to 14

Windows 10 will provide for improved identity protection, data protection, threat resistance, and device 15

security. The Operating System Software refresh will avoid vendor obsolescence, simplify and 16

modernize the desktop office experience,13 enhance our cybersecurity position,14 mobilize SCE’s 17

workforce with “anywhere access,” reduce long term investment and operational costs, and supports 18

more mobile technology capabilities. 19

11 For example, the Microsoft Support Lifecycle policy provides consistent and predictable guidelines for

product support availability when a product releases and throughout that product’s life: https://support.microsoft.com/en-us/lifecycle.

12 An endpoint device is an Internet-capable computer hardware device on a TCP/IP network. The term can refer to desktop computers, laptops, smart phones, tablets, thin clients, printers or other specialized hardware.

13 Windows 10 Simplification and Modernization the desktop experience: Windows 10 will simplify the desktop experience by providing a consistent experience across phones, tablets, and PCs; it will provide current enterprise-grade security to help protect against modern threats; and simplify the management of both corporate and personal devices.

14 Enhance Cybersecurity: Windows 10 offers key architectural changes such as Enterprise Data Protection (EDP) which provides a strong foundation for some of the key data loss prevention capabilities that SCE needs. Its capabilities will be extended with the Rights Management Services (RMS) that comes with Office 365. By using both EDP and RMS, SCE will be able to strengthen protection by limiting basic copy-and-paste when appropriate and prevent printing and forwarding of documents without authorization.

10

In addition, upgrading these operating systems to Windows 10 will avoid future 1

costs related to hardware refresh, software licenses, and maintenance upgrade costs by simplifying the 2

environment and avoiding expensive custom support agreements typically required for software that is 3

no longer supported under standard agreements. 4

Database platform updates will update the database software and perform related 5

application testing and remediation. Regular updates are required to address software compatibility 6

issues, to scale the installation of the database platform in consideration of data growth and usage 7

growth, and to comply with the vendor support life cycle for that product. This must maintain acceptable 8

performance according to product vendor requirements, which specify their supported operating system 9

versions and minimum server hardware capabilities. This will provide product stability, reduce costs by 10

avoiding the need for custom support agreements for software used beyond its supported life cycle, and 11

avoid technology obsolescence. 12

The Business Intelligence (BI) Tools Upgrade will update infrastructure and tool 13

software as required to continue operating our BI capabilities. Regular upgrades are required to address 14

software issues and scale the installation of the BI applications based on data growth and usage. This 15

will allow for optimal performance of the BI applications used across the organization. 16

As the BI software products and services continue to evolve, existing capabilities, 17

such as cloud, mobile, data discovery, data visualization, and predictive capabilities, are enhanced to 18

provide new and improved functionalities. As new functionalities are introduced, SCE must keep the 19

existing products current to meet the demands required for business decision support and analysis. 20

Failure to maintain the products with upgrades puts the organization at risk of technological 21

obsolescence, unsupported software versions and increased custom maintenance support agreements. 22

As integration tools are used by more applications, the size of the environments15 23

and product versions must be reviewed regularly to verify we have the right capacity. SCE needs to keep 24

the environment current to maintain acceptable performance, comply with vendor support life cycle 25

(avoiding higher maintenance costs), provide reliability and stability of the tools, avoid technology 26

obsolescence, and facilitate new innovations and capabilities to drive better decision making. 27

15 An “environment” refers to where computer users run application software.

11

In the Enterprise Platform Core Refresh we will conduct a periodic major version 1

update (also referred to as “refresh”), pursuant to our software license contract with SAP. The last 2

version update of the software was during 2012 and 2013. This refresh is planned for 2017 and 2018. 3

The Enterprise Platform Core Refresh will require system analysis, modifications to eight existing SCE 4

SAP related custom software packages, over 500 interfaces to other SCE systems, testing, and 5

implementation. 6

As with most COTS, SAP periodically updates its software by releasing support 7

packs and enhancement packs. SAP releases support packs and enhancement packs every year. Support 8

packs deliver fixes for problems in the software reported by SAP product customers and specialized 9

computational changes (e.g., legal-related updates needed to align with changes in the federal tax laws). 10

SAP recommends that customers apply support packs upon release to keep the installed software 11

current. Enhancement packs change the software system’s functions such as restructuring the system 12

software modules for technical efficiency (e.g., moving the employee succession planning function from 13

the recruiting application to the main application for human capital), adding functions to meet regulatory 14

demands (e.g., adding the international accounting standards required for U.S. businesses), and adding 15

new business functions based on client demand. The general software industry practice advises staying 16

within two versions of the currently-available product version to maintain support from the vendor, and 17

to mitigate risks due to security problems and failure because of vendor obsolescence and technology 18

obsolescence. 19

SCE has been implementing SAP support and enhancement packs together every 20

other year since completing our SAP installation in 2008. Although SAP releases new versions of its 21

software every year, SCE has implemented a support and enhancement pack update every other year for 22

minor enhancements and software fixes, and a major core refresh to a new version of the software every 23

four years. This strategy results in our not installing the newest versions, which may contain software 24

bugs (common with new software products) that will be fixed in later releases of these versions. This 25

also minimizes our risk of technology obsolescence and is a cost effective way to minimize labor 26

installation costs. 27

3. Scope and Cost Forecast 28

a) Recorded Expenditures 29

Figure II-2 shows our 2015 request, authorized and recorded amounts for 30

operating software. 31

12

Figure II-2 2015 Request, Authorized and Recorded Expenditures

Operating System Software (Nominal $Millions)

In 2015, SCE spent more than authorized on Operating System Software as a 1

result of needing to upgrade our database software to Hana. Procurement of Hana was needed to avoid 2

extending the life of Teradata beyond end of 2017, which would have resulted in increased O&M and 3

hardware expenses. These costs were in addition to our normal server, storage and desktop operating 4

systems environments prior to vendor published end-of-life support for the Microsoft, Cisco, and 5

VMware Enterprise License Agreements and in preparation for our migration to Microsoft Office 365. 6

b) Forecast Expenditures 7

Total forecast expenditures are $82.6 million for operating software updates in the 8

2016 – 2020 period. This forecast includes the cost to update the software, update the underlying 9

operating system software where required, and conduct application testing and remediation. This five-10

year forecast represents a substantial reduction from the previous five years (2011 – 2015) expenditure 11

level of $177 million. This results from a concerted effort to use more virtualization technologies,16 and 12

16 Using virtualization allows several operating systems running in parallel on a single central processing unit

(CPU). This parallelism tends to reduce overhead costs.

$15.67 $15.67 $15.67

$29.93

$14.27

2015 Request 2015 Authorized Variance 2015 Recorded

13

attempts to lower our overall cost structure by leveraging cloud and software-as-a-service subscriptions. 1

An example of this is the implementation of the Microsoft productivity software Office 365,17 which 2

replaced software previously running in our data centers. 3

The capital forecast for this refresh program was developed using SCE’s internal 4

cost estimation model. This model utilizes industry best practices and SCE subject matter expertise to 5

estimate project cost components. SCE’s forecast for this project includes costs for software, hardware, 6

licenses, vendor labor, and SCE IT labor. See this project’s workpaper for the cost breakdown 7

information. 8

For Operating System (OS) software, we forecast $5.6 million to update Windows 9

Server 2003 and software management tools in 2016. In 2017 we forecast $6.1 million to update server 10

operating system software, including Guardium and other Microsoft server products. In 2018 we 11

forecast $7.8 million to update server operating system software and $5 million to complete Windows 12

10 application remediation. In 2019 we forecast $10.9 million to update server operating system 13

software and $10 million for Windows 10 deployment. In 2020 we forecast $6.6 million to update server 14

operating system software and $5 million to complete Windows 10 deployment. 15

For database system software, we forecast $2.5 million in 2016 to update our 16

Oracle Exadata environment, and $2.0 million in 2020 to replace the System D data warehouse. 17

For business intelligence, we forecast $0.3 million in 2017 and $0.5 million to 18

update SAP BusinessObjects suite, IBM Data Stage, and Extract Transfer and Load tools used in SCE 19

analytics platforms for data ingestion and transformations. We forecast $1.0 million in 2019 to update 20

SAP Hana Geospatial Analytics and Predictive Analytics software. 21

For integration tools software, we forecast $0.3 million in 2017, $1.0 million in 22

2018, and $1.0 million in 2019 to update Managed File Transfer (MFT) software used for Business-to-23

Business (B2B) integration; IBM WebSphere Enterprise Service Bus used in SCE.com; update IBM 24

WebSphere Application server; and update IBM SAP Process integration software used for application-25

to-application integration. In 2020 we forecast $0.3 million to update IBM Data Power software used as 26

a security gateway for business-to-business integration. 27

17 Office 365 is the brand name used by Microsoft for a group of software and services subscriptions providing

productivity software and related services running in the Cloud.

14

For the SAP Enterprise Platform Core Refresh, SCE expects to perform analysis 1

in late 2016 and complete the installation of the major refresh over a 15-month period in 2017 and 2018. 2

We forecast $16.70 million to complete this update. 3

15

III. 1

CYBERSECURITY & IT COMPLIANCE 2

The importance of cybersecurity to the utility industry and SCE has expanded as systems and 3

data have become more integral to business operations and as the electric infrastructure has become 4

more essential to national commerce and communications capabilities. Cyber-attacks are continually 5

growing in number and sophistication, and the availability of cyber weapons is on the rise as well. 6

Therefore, maintaining a strong defense against cyber-attack requires a continually evolving set of 7

strategies. Recent examples of cyber-attacks are well documented in the news media and the intelligence 8

community, which include but are not limited to: 9

• Anthem data breach resulting in losing 80,000,000 sensitive personally identifiable 10

information (PII)18 data records (February 2015).19 11

• Sony Pictures America resulting in loss of systems, data, and services with remediation costs 12

estimated in the tens of millions (December 2014).20 13

• EBay loss of a large database of user credentials (March 2015).21 14

• US Office of Personnel and Management resulting in losing sensitive data related to security 15

clearance holders (June 2015). 16

• JP Morgan Chase loss of 76,000,000 personal information records (July 2014).22 17

• Disruption of Ukraine power grid causing over 225,000 customers to lose power. (March 18

2016)23 19

Cyber-attacks are being mounted against an array of organizations resulting in significant impact 20

to customer data privacy and system availability. To protect the data privacy of our five million 21

customers, and the integrity of our critical grid infrastructure, it is imperative that SCE implement strong 22

cybersecurity controls to mitigate risk. 23

18 Personally Identifiable Information (PII), as used in US privacy law and information security,

is information that can be used on its own or with other information to identify, contact, or locate a single person, or to identify an individual in context.

19 Refer to WP SCE-04, Vol. 2 Bk A pp. 44-46. 20 Refer to WP SCE-04, Vol. 2 Bk A pp. 47-50. 21 Refer to WP SCE-04, Vol. 2 Bk A pp. 51-53. 22 Refer to WP SCE-04, Vol. 2 Bk A pp. 54-57. 23 Refer to WP SCE-04, Vol. 2 Bk A pp. 58-63.

16

Figure III-3 SCE Detected Intrusion Attempts

Figure III-3 illustrates the number of intrusion attempts24 on the SCE network over the past five 1

years. As these threats significantly increase year over year, the SCE must develop stronger defense 2

strategies. SCE employs a defense-in-depth cybersecurity strategy,25 which uses multiple layers of 3

protection to prevent unauthorized access to its systems. SCE invests in several Cybersecurity capital 4

programs. These projects fall into three primary categories: Perimeter Defense, Interior Defense, and 5

Data Protection. 6

• Perimeter Defense: Perimeter Defense includes the processes, procedures, personnel, 7

hardware and software designed to protect SCE’s information and systems from external 8

attacks. Like security defense in a home, one can think of Perimeter defense as the locks on 9

doors and windows. Basic technologies include firewalls and intrusion detection systems. 10

24 An intrusion attempt is defined as an unauthorized attempt to access a system, network or endpoint. 25 Refer to WP SCE-04, Vol. 2 Bk A pp. 64-108.

17

The perimeter defense technology prevents, protects, and detects attacks reducing the risk to 1

critical back-end systems. Perimeter Defense is especially critical to systems that are 2

accessible via the Internet. 3

• Interior Defense: The goal of the Interior Defense program is to secure SCE’s internal 4

business systems from unauthorized users, devices, and software. Interior defense is like a 5

video surveillance system in a home, to know where people are and what they are doing in 6

case we need to respond to an emergency. Advanced and integrated real-time monitoring of 7

SCE’s internal business network makes it more difficult for unauthorized users to gain access 8

to our systems and for rogue devices or software to cause business disruption. 9

• Data Protection: The objective of the Data Protection program is to protect SCE customers, 10

employees, contractors, and other personnel from identity theft. The program also protects 11

confidential SCE information residing on all computing devices from unauthorized use, 12

distribution, reproduction, alteration, or destruction. 13

• SCADA Cybersecurity: SCE must enhance cybersecurity infrastructure services to address 14

emerging advanced cybersecurity threats against Industrial Control Systems.26 The objective 15

of this effort is to implement enhanced cybersecurity controls for SCADA systems and their 16

infrastructure to address modern cybersecurity threats targeting grid control systems. 17

Although in previous filings this section was titled “NERC CIP,” SCE implemented 18

additional SCADA protections with a portion of the approved NERC CIP funding. 19

26 Industrial Control System (ICS) is a general term that encompasses several types of control systems used to

automate industrial processes, including supervisory control and data acquisition (SCADA) systems, distributed control systems (DCS), and other smaller control system configurations such as programmable logic controllers (PLC). These systems are used in a variety of critical applications and industries including energy and utilities, transportation, health, manufacturing, food and water.

18

Figure III-4 Perimeter Defense, Interior Defense & Data Protection

Authorized vs Recorded27 (Nominal $Millions)

As shown in Figure III-4, the Commission authorized SCE to invest $26 million in these 1

cybersecurity programs. SCE recorded $28 million, $2 million above authorized levels, to implement 2

needed cybersecurity tools and controls. This advanced technology purchase replaced a less effective 3

incident response tool to help SCE analysts and Incident Response teams better detect campaigns or 4

attacks on SCE computing endpoints. 5

27 Totals include authorized expenditures for Solutions for Emerging Legislative Mandates program.

$2 $26 $26

$28

0

5

10

15

20

25

30

2015 Request 2015 Authorized Variance 2015 Recorded

19

A. Perimeter Defense 1

Table III-3 Perimeter Defense28

Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)

1. Program Description 2

Perimeter Defense is the first line of defense against cyber-attacks. It is the outer layer of 3

protection for our Defense in Depth approach to cybersecurity, which includes the processes, 4

procedures, hardware, and software to protect critical systems such as SAP, customer data, and 5

ultimately our grid. Today, perimeter security is used to protect our back-end systems from unauthorized 6

access. When properly configured, the perimeter defenses should only permit those activities required to 7

conduct business. Using a perimeter defense security model, the perimeter technology prevents, absorbs, 8

or detects attacks reducing the risk to critical back-end systems. Cybersecurity perimeter defenses 9

include technologies such as firewalls, intrusion detection systems (IDS), application proxies and virtual 10

private network (VPN) servers. Therefore the best network security based on best practice is a layered 11

Defense in Depth approach. This approach includes implementing security in layers with each layer 12

providing an increasing level of restrictive controls. Perimeter Defense is especially critical to systems 13

that are accessible via the Internet. 14

2. Need for Program 15

The energy sector is under continuous cyber-attack,29 and the attack methods, exploits, 16

and capabilities are constantly evolving as new types of attacks are discovered. As referenced in Figure 17

III-3, intrusion attempts against SCE continue to increase. Such attacks include computer viruses, 18

worms, phishing, spyware, and advanced persistent threats, any of which could cause significant damage 19

to SCE’s information systems, if successful. Security Magazine writes, “The modern enterprise network 20

has become expansive, porous, and completely blurred due to the large number of Internet-facing 21

28 Refer to WP SCE-04, Vol. 2 Bk A pp. 109-120. 29 Refer to WP SCE-04, Vol. 2 Bk A pp. 115-116.

CIT-00-TR-RM-000002 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Recorded / Forecast 3.64 4.04 7.06 4.83 12.19 11.77 12.80 13.00 13.50 13.50 13.50

Recorded Forecast

20

applications that have been deployed and adopted. The number of potential entry points into the 1

enterprise network has proliferated uncontrollably.”30 As SCE’s enterprise network expands and 2

integrates with more and sophisticated technologies, SCE must deploy advanced perimeter defense 3

technology to keep pace. Without these defenses, systems could be vulnerable to a wide variety of zero-4

day infections (which are previously unknown malware for which an antivirus mitigation is not yet 5

available) and could be accessed by anyone on the internet with malicious intent. 6


This project will increase the security of our controls that prevent unauthorized access to 8

the business systems and data within our internal business network. This project will implement tools 9

that increase remote access security on SCE and employee-owned devices, such as cell phones, laptops, 10

etc. In addition, SCE will continue to implement next generation intrusion protection (such as firewalls), 11

and intrusion detection systems (such as advanced data analytics capabilities), to improve detection of 12

nefarious activity. This project will also integrate these new tools and controls into our existing 13

perimeter defense layer to create a common, unified monitoring platform that allows for rapid response 14

to security events. 15

SCE is requesting $66.3 million for the 2016 – 2020 period to enable this scope of 16

work.31 The capital forecast for this project was developed using SCE’s internal cost estimation model. 17

This model utilizes industry best practices and SCE subject matter expertise to estimate project cost 18

components. SCE’s forecast for this project includes costs for software, hardware, licenses, vendor 19

labor, and SCE IT labor. 20

30 Refer to WP SCE-04, Vol. 2 Bk A pp. 117-120. 31 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure

process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.

21

B. Interior Defense 1

Table III-4 Interior Defense32



Whereas Perimeter Defense acts as a home’s gates, alarm systems, and locks, Interior 3

Defense acts as a video surveillance system that is focused on where people are and what they are doing 4

in the home. Interior Defense is a set of protection controls that are necessary to secure SCE’s internal 5

business systems from unauthorized users, devices, and software attempting to access SCE’s business 6

systems, and to utilize analytics to prevent attacks from happening before they start. These efforts are 7

also focused on identifying and blocking security breaches from personnel with authorized access to the 8

systems. Users of SCE’s business systems can propagate and/or launch malware33 knowingly or 9

unknowingly. Without these controls, SCE could not identify or react to an infected or malicious 10

computer attempting to infect others on the network. Early identification of suspicious activity will 11

allow us to take quicker action to minimize any potential damage that may result from interior attacks. 12


There are several significant changes in the business environment that create new 14

cybersecurity risks that we must defend against. 34 These changes include: 15

• Growth in mobile technologies, which provides more entry points for vulnerabilities 16

to enter our environment. 17

• Growth in cloud computing, which expands our network to environments we do not 18

directly control and which could be hosted on someone else’s network. 19

32 Refer to WP SCE-04, Vol. 2 Bk A pp. 121-126. 33 Malware is software that is intended to damage or disable computers and computer systems. 34 Refer to WP SCE-04, Vol. 2 Bk A pp. 127-131.

CIT-00-TR-RM-000004 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020


Recorded Forecast

22

• Use of business and personal social media platforms, which if not monitored could 1

prevent us from seeing insider threat behaviors, or from identifying and preventing 2

employees from providing information to potential adversaries who would use the 3

information to attack our network. 4

• Growth in the number of internet-connected devices (Internet of Things), which 5

increase the potential attack paths/exposures to our network. 6

• Integration of software platforms and modular applications, which can create new 7

gateways for malicious activities to infect devices. 8

• Growth of insider threats, which is an ever increasing risk and popular attack path 9

that requires SCE to expand our technology to assess user behaviors. 10


This program will enable advanced and integrated real-time monitoring of SCE’s internal 12

business network, which will make it difficult for unauthorized users to gain access to our systems, and 13

for authorized users to knowingly or unknowingly propagate cybersecurity attacks. It will also make it 14

more difficult for rogue devices or software to access SCE systems and confidential data or cause 15

business disruption. This program will also address Advanced Persistent Threats (APT)35 by using 16

advanced data collection and analysis technologies that can provide early detection of potential 17

questionable activity. 18

To accomplish this, the Interior Defense program will perform the following activities: 19

• Extend SCE’s Identity and Access Management system to newer generation security 20

technology; 21

• Enhance and expand SCE’s data collection capabilities to mine and potentially 22

connect disparate pieces of data to form a clear picture; 23

• Implement technology to allow SCE to analyze collected information for security 24

threats in a more automated and effective manner; 25

• Initiate automated alerts when questionable activity is detected to enable us to stay 26

ahead of possible threats and help prevent attacks from happening. 27

35 Advanced Persistent Threats (APT) are a network attack in which an unauthorized person gains access to a

network and stays there undetected for a long period of time. The intention of an APT attack is to steal data rather than to cause damage to the network or organization.

23






C. Data Protection 6

Table III-5 Data Protection37



The Data Protection Program safeguards SCE’s core information computing 8

environment38 by introducing controls to protect critical business information. This program will protect 9

SCE customers, employees, contractors, and other personnel from identity theft, and protect confidential 10

SCE information residing on all computing devices from unauthorized use, distribution, reproduction, 11

alteration, or destruction. This program will improve the security of information stored within various 12

databases both within and outside of SCE’s computing environment. 13

The Data Protection Program will add specialized technology that will: 14

• Increase protection and encryption of data fields within files; 15

• Protect business information on mobile devices; 16

• Enhance access controls to protect sensitive business information; 17

36 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure


37 Refer to WP SCE-04, Vol. 2 Bk A pp. 132-137. 38 SCE’s core information computing environment includes customer data in Customer Service databases, back-

end systems for SCE.com, and SAP, which contains employee data.

CIT-00-TR-RM-000003 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020


Recorded Forecast

24

• Control the replication of business information to personal removable storage media; 1

and 2

• Protect business information stored at external sites hosting SCE business systems. 3


As discussed at the start of Chapter III, several large scale data breaches impacted large 5

and established companies in 2014, including Sony, JP Morgan Chase, Target, and eBay. These data 6

breaches resulted in significant financial, reputational and proprietary impacts to these companies. These 7

companies will be forced to spend hefty funds on improved security measures by way of consultants, 8

security vendors, and test runs—not to mention the fees for lawyers, pending lawsuits, and paying fines 9

from data protection authorities. Given the magnitude of these threats, it is imperative for SCE to 10

implement ongoing cybersecurity improvements to data protection. 11


This program will perform the following activities to achieve the objectives described in 13

the Project Description section: 14

• Implement enhanced controls for granular data protection by deploying Data Loss, 15

Categorization, and Identification tools; 16

• Automate data classification by tying together the different systems that contain data 17

and the ability to classify them; 18

• Monitor and alert unauthorized access to business information by leveraging the 19

monitoring and data analysis environment with new toolsets; 20

• Manage business information saved on personal devices through implementation of 21

more robust endpoint and mobile device tools; and 22

• Control the copying of business information to removable devices such as memory 23

sticks and DVDs. 24

SCE forecasts $27.5 million for the 2016 – 2020 period to enable this scope of work.39 25

The capital forecast for this project was developed using SCE’s internal cost estimation model. This 26


process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding

(Continued)

25

model utilizes industry best practices and SCE subject matter expertise to estimate project cost 1



D. SCADA Cybersecurity 4

Table III-6 SCADA Cybersecurity40



This project extends security measures beyond the NERC CIP requirements governing 6

the bulk electric system by implementing additional risk-reduction methods specifically for SCE’s 7

Supervisory Control and Data Acquisition (SCADA) systems. SCE’s SCADA systems provide remote 8

control and monitoring capabilities of the electric grid. While NERC CIP requirements establish a 9

security-control baseline, SCE believes that the rise of electric grid threats and a strong understanding of 10

SCE’s specific environment warrants implementing further security control measures. 11

SCADA Cybersecurity should not be confused with SCE’s Grid Modernization 12

Cybersecurity project presented later in this testimony. SCADA Cybersecurity involves protections for 13

legacy industrial control systems that are currently connected via routable networks. Improved visibility, 14

detection, and protection controls are needed to secure these environments from the evolving threats that 15

continue to propagate against the utility industry. Grid Modernization Cybersecurity efforts are centered 16

on implementing security controls for systems that are interconnected to each other or to external 17

devices such as distributed energy resources (DERs). 18

Continued from the previous page the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.


CIT-00-TR-RM-000017 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 1.06 0.09 0.15 1.26 0.38 - 8.80 8.95 2.50 2.50

Recorded Forecast

26

SCE has invested in visibility tools and firewalls designed for grid segmentation to create 1

choke points that enable us to close off certain areas of the network in the event of an attack. It also 2

allows us greater visibility. These firewalls act like a watch post at a guarded facility checking 3

identification to validate that you are who you say you are. The goal of this current effort is to focus on 4

additional SCADA protection needs, taking a programmatic approach to reduce the risk. The approach 5

includes performing an assessment of current state cybersecurity controls, establishing desired target 6

states based on advanced cybersecurity risks, and implementing enhanced controls commensurate with 7

system risk. Changes to these systems will be used to update and enhance cybersecurity incident 8

response and investigation capabilities by improving and maturing controls or adding new ones. In some 9

cases, military grade technology will be needed to defend against nation-state attacks, which may 10

require additional implementation costs. 11


The energy sector has become a major focus for targeted attacks and is now among the 13

top five most targeted sectors worldwide.41 SCADA systems are increasingly becoming the target of 14

sophisticated cybersecurity attacks as evidenced by recent Black Energy attacks on Ukrainian power 15

grids.42 These systems are more vulnerable to cyber-attack as they have longer refresh cycles, fewer 16

security updates, 100% expectation of reliability to our customers, and require significant coordination 17

to test and implement security upgrades. 18

In the last three years SCE has seen a ten-fold rise in attempted cybersecurity intrusions. 19

Coupled with warnings from James Clapper, Director of National Intelligence, that advanced attacks 20

against the electric grid are an imminent threat, SCE believes that its customers are best served by 21

implementing additional SCADA security controls. Beginning in 2011, while implementing NERC CIP 22

controls, SCE began enhancing SCADA security controls as SCADA concerns arose, and resources 23

permitted. 24

The threats and challenges against the grid underscore the need to further enhance 25

cybersecurity for SCE’s SCADA systems. Below are infrastructure services that SCE believes the 26

SCADA systems should leverage to garner more robust security controls. 27

41 Refer to WP SCE-04, Vol. 2 Bk A pp. 141-170. 42 Refer to WP SCE-04, Vol. 2 Bk A pp. 171-173.

27


The SCADA cyber security project scope includes the following: 2

• Build a secure network to protect the administrative interfaces of critical tools; 3

• Develop device access controls to secure how operators interact with control systems; 4

• Develop user access controls to secure role-based access to least-required 5

privileges,43 which is a more secure profile for user access; 6

• Implement next generation malware protections to identify malware in an 7

environment; 8

• Deploy vulnerability management tools to scan the environment looking for known 9

vulnerabilities; 10

• Provide data encryption services to encrypt data at transit and at rest; 11

• Develop system monitoring services to provide greater visibility to the network; 12

• Implement threat intelligence integration tools that can automatically ingest 13

intelligence to monitor and analyze the environment; and 14

• Procure government-sponsored secure technology to defend against advanced attacks. 15






43 The Principle of Least Privilege is the idea that only the most minimum number of people should have access

to information and resources that are necessary for its legitimate purpose. 44 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure


28

E. Common Cybersecurity Services for Generator Interconnections 1

Table III-7 CCS for Generator Interconnections45



In SCE’s 2015 GRC, the Commission adopted the Common Cybersecurity Services 3

(CCS) for Generator Interconnections project.46 This project enables the design and enforcement of 4

policies that can be configured for a type of SCADA system in the electric grid. Each device on the 5

electric grid secured by CCS will have a unique key to enable secure communications with its control 6

system. This approach mitigates the risk that an attacker can seize control of the electric grid from an 7

individual device, such as a relay or capacitor bank controller, and provides the ability to rapidly 8

respond to a cybersecurity event. 9

This project will deploy a Central Security Services engine47 and Edge Security Services 10

systems48 to protect critical electricity generator interconnections. The Central Security Services engine 11

consists of two services that are physically located at SCE’s grid control centers. These include: 12

• Central Security Configuration Services, which manages secure encrypted 13

connections and system health checks on CCS member systems. 14

• Automated Security Services are control actions defined in the configuration that 15

execute when a given set of events occur. 16

45 Refer to WP SCE-04, Vol. 2 Bk A pp. 174-179. 46 This project was titled “Common Cybersecurity Services (CCS)” in SCE’s 2015 GRC. 47 A Central Security Services engine is the central system used to manage the secure connections created by

CCS. 48 Edge Security Services systems are located at the edge of grid networks that provide measurement or control

data back to central SCADA systems.

Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - 6.86 8.15 9.78 7.36 1.00 3.40 5.90 - - 42.46 2015 GRC Authorized/Request* - 6.86 8.15 5.34 8.23 8.46 8.68 45.72 2015 GRC - Original Request - 6.86 9.90 5.34 8.23 8.46 8.68 47.47 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 and 2015 levels. 2016 and 2017 reflect forecast amounts from SCE's 2015 GRC.

Multiple CITs: CIT-00-DM-DM-000141, CIT-00-SD-PM-000103, and CIT-00-SD-PM-000175

Recorded Forecast

29

The Edge Security Services systems consist of services that provide distributed 1

enforcement of cybersecurity on devices at or near the perimeter of a SCADA system. This protects 2

edge configurations from being altered in a manner that introduces cybersecurity risk to central control 3

SCADA systems. These systems are also supported by the Automated Security Services engine to take 4

automated responses when configuration events are triggered that are deemed a risk to the overall 5

system. 6

2. Need for Project 7

The Common Cybersecurity Services (CCS) project will provide enhanced cybersecurity 8

protections for critical generator interconnections. The applications on these interconnection paths 9

require low latency49 to transmit data to back-office systems. It is critical to maintain assurance over 10

these network paths as these systems make automated control decisions on the electric grid. The CCS 11

system is specially designed to provide cybersecurity assurance over these paths while maintaining the 12

minimum performance requirements to enable the functionality of low latency control systems. This 13

system provides controls to meet critical NERC CIP compliance requirements as it relates to electronic 14

security perimeters. 15


CCS for SCE’s Phasor system was implemented in 2015. This implementation used a 17

proprietary vendor solution. After conducting an assessment on the cost effectiveness of the current 18

version of CCS on the Phasor system, SCE decided that due to the high device integration costs, CCS is 19

too cost prohibitive to scale in its current form. Consequently, SCE will invest in a non-proprietary, 20

standards-based version of CCS that can scale to meet the security needs of the bulk electric system. 21

SCE believes that this revised approach is in the best interests of our customers and the security of the 22

electric grid. 23

SCE’s original, adopted forecast for this project was $47.5 million. Our revised total 24

project forecast through the end of 2017 is $42.5 million, of which $33 million has been incurred 25

through the end of 2015. This total project cost reduction results from scope modifications that include: 26

49 Low latency refers to systems that require having a very low time interval between when a message is sent

and when it is received.

30

• CCS security for EMS and SCADA will be scheduled separately as part of the 1

network upgrades and cybersecurity implementations associated with SCE’s Grid 2

Modernization program;50 and 3

• The refresh of CCS central services servers that can be deferred until 2021. 4

SCE is requesting $10.3 million for the 2016 – 2020 period to complete this project.51 5





F. Grid Modernization – Cybersecurity 10

Table III-8 Grid Modernization Cybersecurity52



Grid Modernization enables new capabilities to support the evolving use of the 12

distribution system. This will require many new applications that extend grid networks in a two-way 13

50 Refer to SCE-02, Vol. 10 for more information on SCE’s Grid Modernization program. Cybersecurity related

efforts during the 2016-2020 period will be addressed in the Grid Modernization Cybersecurity program within this testimony.

51 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.

52 Refer to WP SCE-04, Vol. 2 Bk B pp. 180-186.

CIT-00-TR-RM-781701 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 5.28 16.19 24.44 29.15 24.80 99.86

Previous GRC Request - - - - - - - -

Recorded Forecast

31

relationship with customers and third parties. The distributed intelligence53 from grid modernization 1

presents new cybersecurity challenges to the grid of the future: 2

• Two-way communications with edge devices open new avenues of attack as edge 3

computing devices will communicate with control centers over routable connections 4

in near real-time. 5

• Implementation of secure network segmentation and survivability strategies must 6

protect grid reliability if a cyber-attack occurs. 7

• Integration of new technologies with cybersecurity infrastructure. 8

• Significant upscaling of cybersecurity service layers to automate controls, 9

monitoring, and management for new grid operations. 10

• Protection of legacy systems that do not support modern security protocols. 11

• Enhanced security for system monitoring and management networks with privileged 12

access to control systems. 13

Addressing these cybersecurity challenges requires a combination of infrastructure, 14

application, and threat intelligence initiatives. Infrastructure service layers54 are needed to extend strong 15

cybersecurity controls to edge networks. New grid applications must be designed with cybersecurity 16

controls throughout their lifecycle by integrating strong access controls, secure communications, and 17

secure programming code. Integration of cybersecurity operations with external threat intelligence 18

sharing organizations55 will enable more robust incident response and investigation capabilities. 19

Cybersecurity needs to be integrated into each grid modernization component throughout its lifecycle to 20

provide a strong framework against a cyber-attack. Therefore, this project will implement the requisite 21

cybersecurity controls for the various Grid Modernization related hardware and software applications 22

discussed in SCE-02, Volume 10 – Grid Modernization.56 23

53 Distributed intelligence refers to gathering distributed energy resource data points to make power control

decisions on the electric grid. 54 Infrastructure service layers are the systems used to provide cybersecurity device access management, user

access controls, malware protection, vulnerability management, system monitoring, incident investigation and response, and data protection.

55 These organizations include government law enforcement and intelligence, private sector intelligence feeds, and electric industry specific cybersecurity intelligence sharing organizations.

56 See SCE-02, Vol. 10 – Grid Modernization for detailed descriptions of these hardware and software applications.

32

This Grid Modernization Cybersecurity project is distinct from the SCADA 1

Cybersecurity project requested in preceding testimony. Grid Modernization Cybersecurity focuses on 2

extending cybersecurity controls to distribution system substations as they join a routable wide area 3

network.57 SCADA cybersecurity is focused on enhancing cybersecurity controls for critical control 4

center systems performing central management of the bulk electric system. 5

The Grid modernization project is also distinct from the Substation Automation 3 project. 6

The SA3 project leverages existing cybersecurity services to provide enhanced security to managing 7

intelligent electronic devices. SA3 is specifically focused on secure access and management of 8

Intelligent Electronic Devices (IED). It leverages a central cybersecurity service layer to provide 9

enhanced protection to IED access controls and management. 10


New grid capabilities with additional communication channels increase the potential for 12

cyber-attacks. Enabling a distributed intelligence system requires real-time communications from edge 13

distribution systems to central control centers. This upstream data flow will be used to make automated 14

control system decisions that can significantly impact reliability on the electric grid. These systems can 15

be used as a foothold by an attacker to attempt to compromise various layers of the grid network. 16

However, the new communication paths provided by the wide area network will enable a centrally 17

managed cybersecurity controls designed in a more preventative and automated architecture. This 18

architecture will be designed to provide layered defense-in-depth cybersecurity controls while enabling 19

distributed intelligence systems. 20

Despite the implementation of strong preventative controls, cybersecurity design must 21

account for the possibility that a compromise of a system on the distribution network will occur. A 22

compromised system on the grid enables an avenue of attack to escalate privilege, launch malware 23

attacks, or render a grid system inoperable. Preventative controls alone will not adequately mitigate the 24

potential cybersecurity risk of a malicious insider or sophisticated attacker. The cybersecurity 25

protections must be able to identify when a compromised system behaves anomalously and execute an 26

automated response to isolate the system to minimize its potential impact to the grid operations. 27

57 A routable wide area network is a geographically dispersed telecommunications structure that allows

communications between a series of local networks. I.E. network connectivity between substations and control centers that can be routed per the needs of the grid modernization applications.

33

Implementation of these types of controls requires advanced monitoring and behavior analysis to 1

identify and prevent system exploitation. A concerted initiative is required, so cybersecurity technology 2

and best practices are implemented throughout our grid modernization effort. New two-way 3

communication paths and grid management applications must operate on secure cybersecurity 4

infrastructure and be designed with cybersecurity capabilities and requirements throughout their 5

lifecycle. 6


SCE forecasts $99.9 million for the 2016 – 2020 period to complete this project. The 8

capital forecast for this project was developed using SCE’s internal cost estimation model. This model 9

utilizes industry best practices and SCE subject matter expertise to estimate project cost components. 10

SCE’s forecast for this project includes costs for software, hardware, licenses, vendor labor, and SCE IT 11

labor. See this project’s workpaper for the cost breakdown information. This forecast will support the 12

implementation of the functions and technologies discussed below.58 13

Cybersecurity for the power grid must be carefully engineered not to interfere with 14

energy delivery functions. For instance, our power grid has legacy devices that are decades old, with 15

limited computational resources and communications bandwidth to support cybersecurity protections. 16

Control and protection devices are widely distributed; some are in unmanned, remote substations or on 17

top of poles in publicly accessible areas. Cybersecurity controls are important and have the potential to 18

disable critical grid systems if configured in an overly strict manner without proper reduction of system 19

false positives.59 Operation of cybersecurity controls must not jeopardize normal operations or 20

emergency responses. Thus, cybersecurity controls must be designed to provide the appropriate response 21

to an alert without disabling critical energy delivery systems. Grid modernization will require significant 22

redesign of cybersecurity architecture, concept of operations, and operationalization planning to 23

facilitate the organizational change management to operations technology and information technology. 24



59 System false positives are a test result which incorrectly indicates that a particular condition or attribute is present.

34

New Grid Modernization applications will enable integration with distributed energy 1

resources and communication relationships with third parties. These new interactions necessitate that 2

development efforts perform specialized secure application coding reviews throughout the development 3

lifecycle to minimize the possibility of introducing new vulnerabilities to grid. This project will 4

thoroughly vet application code through a multitude of secure code review services to identify and 5

remediate vulnerabilities in the architectures, system inputs, cryptographic implementations, access 6

controls, database security, memory management, communication sessions, and system configurations. 7

Secure coding is more challenging as it introduces limitations to application functions that could 8

otherwise be used for malicious purposes. This process will be conducted for the Grid Modernization 9

software applications mentioned above. 10

Grid Modernization also necessitates changes to the distribution grid cybersecurity 11

service infrastructure. This project will provide enhanced segmentation and inspection between 12

distributed energy resources and third parties must be implemented to enforce system separation 13

engineering principles. This must provide maximum isolation of critical service layers and systems from 14

new avenues of communication originating from untrusted systems and networks. 15

Cybersecurity control sensors placed throughout the system will need to work in concert 16

to limit any potential attack. The ability to identify the origin of a breach and segment these systems 17

from the network is paramount to limiting the impact of any given attack. Network segmentation of the 18

system must be implemented in a manner that provides segmentation between control center and bulk 19

systems communications while enabling the new communication paths required for grid modernization. 20

Systems must enforce non-repudiation60 of all user activity to deter insider threat and 21

track system usage. Access controls must be strong enough to uniquely identify each user of a system 22

without preventing access to the system in the event of connectivity loss to central authentication 23

systems. This requires implementation of distributed privileged access management systems to enable 24

auditing of shared control system credentials. This data must all be integrated into centralized security 25

operations monitoring to perform cybersecurity analytics and facilitate incident response. Upon 26

detection of suspicious access behavior the system must notify incident response teams to mobilize a 27

response to potential system access loss. This requires baselining access behavior and alerting upon 28

60 Non-repudiation refers to the concept that a user of a system is unable to deny that they performed a specific

action on that system, i.e., a log attributable to a person’s name is generated for all system activity.

35

unauthorized or irregular use of privileged system access. The system will then be able to create alerts 1

based upon unauthorized user activity. Balancing the benefits of central access control with the needs for 2

local support of systems in the event of network loss is a critical control required to protect the 3

cybersecurity of the system without compromising its reliable operation. 4

This project will also tune monitoring systems to detect anomalous and suspicious 5

activities while permitting new and existing grid applications. Electric grid controls systems operate in 6

predictable communication patterns using a limited number of communications protocols. This project 7

will implement systems that are able to aggregate logs and communication data to detect when a system 8

or systems begin communicating in an anomalous way to detect potential attackers. For example, if a 9

compromised system attempts to launch a denial-of-service attack on the grid it would generate 10

significantly more network traffic than is typical during normal operation. Maintaining an automated 11

baseline of system configurations and communication behavior will enable an immediate and 12

accelerated response to such an attack upon detection of anomalous traffic. It would isolate the origin of 13

the attack, enable automated blocking of denial of service traffic, and segment this portion of the grid 14

from critical systems until remediation is complete. This would significantly impede the effect of this 15

attack and prevent the compromise from spreading to other portions of the grid network. 16

Detection and management of authorized and unauthorized systems and software must be 17

strengthened. Advanced malware detection systems must be tuned to whitelist61 permitted applications 18

and autonomously detect known bad malware and the behavior of unknown malware. Electric grid 19

systems typically do not require frequent software changes. This simplifies the implementation of 20

whitelisting technology to prevent unknown malicious or unauthorized software from executing on the 21

system. This would require an attacker to first circumvent the whitelisting protection prior to executing 22

any malicious code on the system. This significantly increases the difficulty of an attack on a critical 23

system as protection software is integrated into central monitoring systems to detect tampering. 24

Additionally, in the event that malicious code does execute on a system despite whitelisting controls, 25

advanced malware detection systems will be implemented as part of this project to detect malicious 26

behavior and remove the malware from compromised systems. This project will also implement 27

61 Whitelisting involves controls within software that permit known good applications and code to run while

denying all other applications and code from running on a system.

36

malware protection into network security devices to prevent the spread of malware through the network 1

upon detection. 2

The infrastructure changes require tight integration with operations teams to support 3

system availability and coordinated incident response. This project will enable the convergence of 4

operations technology with IT cybersecurity controls for awareness and coordination between system 5

stakeholders to protect the grid from cyber-attack. System behavior data and alerts will be shared with 6

correct stakeholders to enable operational awareness and response. These protections will ingest system 7

configuration and log data into usable analytics dashboards tuned to alert on potential incidents in a 8

timely and accurate manner. These response procedures must be regularly tested with incident exercises 9

and penetration tests to verify the effectiveness of cybersecurity operations. 10

This project will implement secure communication protocols between grid systems by 11

using secure encryption technologies protecting the integrity of control system data. Grid modernization 12

systems will enable automated actions to be taken on the grid based on data from distributed intelligence 13

systems, so cybersecurity infrastructure services will be implemented to provide the services needed to 14

facilitate strong authenticated encryption and key management to enable these new protection 15

mechanisms. The lack of integrity protections on this control data could result in man-in-the-middle 16

attacks where systems take actions based on forged messages lacking cryptographic protection. If the 17

system is responsible for managing power it could be exploited to disable power delivery. This type of 18

protection will also apply to sensitive data leaving the grid. Protections must be put in place to verify 19

that meter and measurement data sent to outside parties has not been tampered with so it may be trusted 20

as it is passed to system stakeholders. This is a critical function to protect power delivery and 21

information sharing of grid data. Grid modernization aims to change the way power is managed and 22

delivered, necessitating adapting robust cybersecurity controls to the changes in grid modernization’s 23

electricity management model. 24

a) Alternatives Considered 25

Grid modernization systems are highly customized and adapted to specific 26

applications using communications protocols that are not traditionally employed in IT environments. 27

Due to this complexity, development of a custom-built cybersecurity control suite was considered to 28

provide a central management and monitoring for all cybersecurity controls. This approach was deemed 29

difficult to scale as no single vendor or software suite was able to provide all of the necessary controls to 30

address the risks. While custom cybersecurity solutions will need to be developed for special purpose 31

37

cases, cybersecurity controls will primarily adapt commercial off the shelf technology to support grid 1

modernization applications. 2

G. IT Support for NERC CIP Compliance 3

Table III-9 NERC CIP Compliance62



This project will continue the on-going implementation of systems and processes that will 5

help SCE maintain compliance with the evolving cybersecurity-related NERC Critical Infrastructure 6

Protection (CIP) requirements. These systems and processes will improve facility access management, 7

asset change control maintenance, and physical access control. In addition, this project anticipates future 8

expenditures associated with emerging mandatory requirements. This project includes cost forecasts for 9

work important to complying with the following FERC-approved requirements: 10

1. NERC CIP-006 – Physical security perimeter requirements for low-impact facilities; 11

2. NERC CIP-014 – Physical security requirements; and 12

3. NERC CIP-010-2 and NERC CIP-007-6 – Change control requirements for numerous 13

asset types. 14

This project forecasts costs required to comply with anticipated FERC requirements 15

applicable to this rate case period. 16


Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 6.18 1.00 0.75 11.79 10.42 14.37 12.92 5.97 3.42 2.47

Previous GRC Request* 6.18 1.00 4.80 14.09 11.00 4.80 5.10 *The Commission adopted SCE's request for 2014 and 2015 expenditures for this program in D.15-11-021.

Multiple CITs: CIT-00-OP-NS-000377, CIT-00-OP-NS-000446, CIT-00-TR-RM-000001, CIT-00-TR-RM-000013, CIT-00-TR-RM-000014, CIT-00-TR-RM-000015,

CIT-00-TR-RM-000016, and CIT-00-DM-DM-000081

Recorded Forecast

38


As discussed in more detail in SCE-08, Volume 1, SCE must adhere to FERC-approved 2

CIP Reliability Standards, or risk being levied penalties or fines.63 Since 2006, NERC has developed and 3

FERC has approved five CIP versions of standards, and subsequently issued several substantive 4

revisions, such as the Revised Critical Infrastructure Protection Reliability Standards often referenced as 5

“CIP v6 Reliability Standards.”64 This project will implement technology-related solutions to enable 6

SCE to maintain compliance with these NERC CIP requirements. 7

As identified in SCE’s NERC Compliance Program section (Article II), SCE must 8

implement approved NERC CIP v5 standards, while FERC continues to evolve the Critical 9

Infrastructure Protection (CIP) mandatory standard requirements. FERC has approved recent NERC CIP 10

v5 revisions and issued orders for CIP modifications. The three NERC CIP requirement areas listed 11

above require SCE’s response to remain in compliance. CIP requirement updates and modifications have 12

been issued consistently since 2006, and SCE has responded in kind. 13

As the past ten years have demonstrated, SCE expects CIP requirements will continue to 14

evolve in this GRC period: requirement gaps will materialize, and threats and technologies will evolve. 15

The following bullets illustrate the continuous pattern of CIP requirement change and substantiate 16

SCE’s reasonable expectation of future changes: 17

• NERC CIP v5 requirements, enforceable beginning July 1, 2016, address the NERC 18

CIP Standard Requirements CIP-002 through CIP-011. The requirements include 19

work such as implementing access controls, change control, and grid system 20

perimeter controls. 21

• FERC approved NERC CIP v5 revisions, from January 21, 2016, are mostly 22

enforceable on July 1, 2016, and largely impact IT.65 Some CIPs have later 23

enforcement dates, such as CIP-003-6, which requires, for low impact Bulk Electric 24

System (BES) Cyber Systems, implementing security plans or plans that address 25

security awareness practices, physical security controls, electronic access controls, 26

and incident response plans; and CIP-010-2, which requires for transient cyber assets 27

63 See SCE-08, Vol. 1, Chapter IV, Section C, O&M for a discussion on applicable fines. 64 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000. 65 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000.

39

and removable media, implementing a plan or plans to include controls over software 1

vulnerabilities mitigation, introduction of malicious code mitigation, removable 2

media authorization, and unauthorized use of transient cyber assets. 3

• FERC ordered NERC CIP modifications that have yet to be drafted and approved.66 4

These modifications include: 5

o Protection of transient electronic devices used at Low Impact BES Cyber System; 6

o Protections for communication network components and data communicated 7

between BES Control Centers tailored to be commensurate with the risk posed to 8

the bulk electric system; and 9

o Modifications to the definition for Low Impact External Routable Connectivity. 10

• FERC created a new CIP, CIP-014, which addresses physical security requirements.67 11

The physical security requirements, however, depend on information technology. 12

• FERC directed NERC to conduct a study that assesses the effectiveness of the CIP 13

remote access controls, the risk posed by remote access-related threats and 14

vulnerabilities, and the appropriate mitigating controls. The results could cause 15

further mandated standard requirements.68 16

• FERC explored the need to develop mandatory requirements for cyber controls in the 17

supply chain, including during a Technical Conference on January 28, 2016. 18


This project includes cost estimates for complying with FERC-approved requirements 20

that remain to be implemented, and cost estimates for complying with anticipated FERC requirements 21

applicable to this rate case period. The FERC-approved controls require work covering the years 2016, 22

2017, and 2018. These forecasts are summarized in Table III-10 below, and follow with a description of 23

the work these forecasts will support. 24

66 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000. 67 See FERC Order No. 802, issued on November 20, 2014, 149 FERC 61,140, Docket No. RM14-15-000. 68 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000.

40

Table III-10 Capital Forecast by NERC CIP Standard

(Nominal $Millions)

First, to support NERC CIP-006 compliance, SCE identified 90 low-impact facilities 1

needing physical security perimeter protection as stated in SCE-07, Volume 5, Chapter V. As NERC 2

CIP required and as the historical expenditures reflect, physical access to high-impact and medium-3

impact facilities has already been addressed. For low-impact facilities, information technology 4

commensurate with the requirements can be leveraged to manage physical access. SCE plans to use 5

cost-effective smart keys to manage this physical access. Similar to hotel card key systems, SCE can 6

program the keys and remotely enable and disable key access. The solution requires performing 7

telecommunications work, which includes installing kiosk and renewal stations in various locations that 8

can connect to SCE’s network, and implementing a smart key application. Capital expenditure is 9

estimated to be $5 million in 2016. 10

Second, to support NERC CIP-014 compliance, SCE has identified nine facilities 11

requiring physical security controls as stated in SCE-08, Volume 1, Chapter V. SCE plans to execute the 12

following related to these nine sites at an estimated cost of $16.39 million covering years 2016 and 13

2017: 14

• Implement new monitoring systems to detect intrusions, including gunshots, seismic 15

events, and other activities that could pose a threat to the facility or BES. 16

• Enhance the Physical Security Information Management system to provide a secured 17

and integrated application allowing Corporate Security personnel to quickly identify 18

and respond when a threat is detected. 19

2016 2017 2018NERC CIP-006 - Physical security perimeter requirements for low impact facilities

5.00$

NERC CIP-014 - Physical security requirements 8.87$ 7.52$

NERC CIP 010-2 and NERC CIP-007-7 - Change control requirements for numerous asset types

3.00$ 1.00$

41

• Upgrade the telecommunications infrastructure to support the new monitoring 1

systems, enabling the transfer of data, alarms, and alerts between the originating sites 2

and the monitoring locations. 3

Third, to support NERC CIP-010-2 and NERC CIP-007-6, SCE must replace the current 4

and interim Test Smart Form Tool (TSFT) tool with a production tool. TSFT is used to collect evidence 5

for all work performed on T&D managed BES Cyber Assets and associated Protected Cyber Assets 6

(PCA). The current tool was never designed to be used in production. Rather, TFST accommodated 7

SCE’s needs while developing new business processes for NERC CIP compliance and to help define 8

requirements for a production tool. The current TSFT has minimal security, does not integrate with other 9

systems, and cannot accommodate the required user base. Due to the temporary nature of the current 10

tool, these shortcomings create an unnecessary compliance risk. SCE estimates capitalized software 11

costs of $3 million in 2017 and $1 million in 2018 for a production tool. 12

Regarding work required for complying with anticipated FERC requirements, IT will 13

continue to have significant responsibility for NERC CIP evolving mandated standard requirements. IT 14

is estimating costs for standard requirements being drafted and for standard requirements FERC will 15

order. To forecast the anticipated costs associated with these future requirements, SCE considered the 16

total IT expenditures for NERC CIP over the 2014 – 2016 timeframe, to forecast future NERC CIP 17

expenditures for 2018, 2019, and 2020. This is shown in Table III-11 below. 18

Table III-11 Forecast Methodology for Emerging IT NERC CIP Requirements

(Constant $Millions)

With evolving mandated standard requirements for NERC CIP, history is the most 19

reliable predictor for estimating future costs during this rate case period. Two types of historical IT-20

spend were excluded when creating the forecast, based on reasonable assumptions: 21

(1) Foundational telecommunication expenditures are excluded because SCE assumes 22

that during this GRC period the NERC CIP requirements will not have the same 23

Expected2014 2015 2016 2018 2019 2020

NERC CIP IT $3.60 $6.76 $0.50 $4.97 $3.42 $2.473-Yr Total

Recorded Forecast

$10.86$10.86

42

magnitude of foundational change affecting telecommunications as they did in the 1

past. 2

(2) CIP-014 expenditures are excluded because a limited spend history exists related to 3

CIP-014, which is a new CIP. 4

Rather than equally allocate the forecasted dollars over 2018, 2019, and 2020, we 5

allocated based on anticipated standards effective dates. 6


For standards drafted and approved, SCE believes the best option is to forecast 8

spend based on the amount and types of work needed to meet requirements. SCE considered whether to 9

forego estimating required expenditures on NERC CIP standard requirements not drafted and approved. 10

SCE considered the probability that standards will evolve and be approved during this General Rate 11

Case cycle to be high. Therefore, SCE has embraced better planning by anticipating that mandatory 12

standard requirements will be drafted, and using historical spend as a predictor of cost, which is 13

illustrated in Table III-11. 14

43

IV. 1

TECHNOLOGY CONSOLIDATION & OPTIMIZATION 2

The Technology Consolidation and Optimization project opportunities include work to 3

standardize and reduce applications across SCE, reduce non-critical hardware, and prioritize IT 4

investments based on business outcomes. The Technology Consolidation and Optimization (TC&O) 5

section includes three enterprise-level projects and six smaller enterprise technology projects under $3 6

million. The following testimonies provide the project description, need, scope, and forecast for 7

consolidation and application rationalization opportunities that will lead to future operating cost savings. 8

A. Data Warehouse Consolidation 9


In 2015, SCE adopted a refreshed and comprehensive enterprise analytics strategy with 11

the goal of simplifying and reducing costs in the SCE technology environment, while providing robust 12

capabilities for data-driven strategic decisions in the fast changing utility analytics landscape. At the 13

core of this strategy is the consolidation of the current SCE data warehouse environments onto an 14

enterprise analytics platform consisting of a foundation of SAP HANA and Cloudera Hadoop 15

technologies. SAP HANA is an in-memory database technology that allows the processing of large 16

quantities of data quickly. Hadoop is an open source software that provides massive storage of any data 17

and the ability to process that data at low costs. With these two technologies as a foundation, the 18

capabilities of the enterprise analytics platform include: 19

• Ability to maximize the use of Hadoop for storage and analytics by leveraging the 20

ability to store large amounts of data at low cost; 21


Table IV-12 Data Warehouse Consolidation69


Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 9.40 4.70 2.00 0.20 0.50 16.80 Previous GRC Request - - - - - - - - Forecasts include CIT-00-DM-DM-000070 and CIT-00-OP-CS-000074

Recorded Forecast

44

• Ability to leverage HANA’s in-memory capabilities for high-performance analytics; 1

• Improved accessibility, consistency, security, and scalability of data in the platform; 2

• Flexibility to cater to different levels of analytics from reporting to statistical analysis 3

to predictive & prescriptive modeling; 4

• Integration with a comprehensive set of existing analytical tools that include the SAP 5

Business Objects Suite, SAS and ESRI to enable ad-hoc analytics, visualization, 6

reports & dashboards, search, data exploration, advanced analytics, etc.; and 7

• Ability to deliver analytics via email or through a web portal on smart phone, tablet, 8

laptop or desktop computer. 9

This project will consolidate the current SCE data warehouse environments onto an 10

enterprise analytics platform and includes the migration of all data and capability off of the current 11

Teradata platform onto the enterprise analytics platform. This includes ensuring all data integration into 12

and out of Teradata are moved to the new platform. Consolidation of other data warehouse platforms 13

onto the enterprise analytics platform will be part of other projects. 14


The migration off of the Teradata Data Warehouse was prompted by the need for further 16

investment in the Teradata platform, requiring a hardware refresh and software upgrade in 2016 to keep 17

the system operational and supportable. This initiative will resolve the issue with the aging Teradata 18

platform by moving to the new platform and will provide additional capability and reduced on-going 19

capital and O&M costs associated with the maintenance of the Teradata platform. It also provides the 20

foundation for the full enterprise analytics platform strategy (consolidation of additional data 21

warehouses onto the enterprise platform), enabling several additional key analytics initiatives. The 22

additional data warehouse platforms planned for consolidation onto the enterprise platform (as part of 23

separate projects) include the Customer Service Data Warehouse (on DB2 mainframe) as described in 24

SCE-04, Vol. 3—CS Re-Platform, Power Procurement Data Warehouse (on Oracle), and the SAP BW 25

Data Warehouse (on a separate instance of SAP HANA). This consolidation is illustrated in Figure IV-5 26

below: 27

45

Figure IV-5 Consolidation onto Enterprise Analytics Platform


To enable this move, SCE made a strategic purchase of SAP HANA in 2015 to support 2

this project and several others. The strategic SAP HANA software contract negotiation allows SCE to 3

purchase 12TB of HANA software and use up to 42TB within 4 years at no additional cost. The benefit 4

case for the strategic purchase of HANA was based on implementing two key projects: (1) The 5

migration from Teradata to the Enterprise Analytics Platform (this project) as part of the overall Data 6

Warehouse Consolidation, and (2) The Core Refresh of the SAP transactional systems, which includes a 7

move onto HANA as the transactional database. While this is the first implementation project to realize 8

the benefits of the strategic purchase of SAP HANA, there are other initiatives that will leverage this 9

HANA capability, including SAP Core Refresh, Grid Modernization, and Customer Service Re-10

Platform. 11

The total project cost forecast is $16.8 million. The capital forecast for this project was 12

developed using SCE’s internal cost estimation model. This model utilizes industry best practices and 13

SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 14

46

includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 1

and hardware costs. See this project’s workpaper for the cost breakdown information.70 2


SCE could potentially continue investing in the Teradata data warehouse 4

platform. This alternative was not chosen due to: 5

• Higher on-going capital and O&M costs for the Teradata platform; 6

• Higher project costs for future analytics initiatives due to the need for 7

increased integration and management of data across multiple platforms; 8

• Reduced capability for analytics, resulting in more complexity and less 9

efficiency for solutions for end-user analytics needs; and 10

• The benefits of moving to the enterprise analytics platform outweighed 11

continued investment in the Teradata platform. 12

B. Lotus Notes Migration 13

Table IV-13 Lotus Notes Migration71



The Lotus Notes Migration project will migrate and consolidate the application 15

functionality or retire 230 Lotus Notes complex and custom business solutions to achieve cost savings. 16

This project will eliminate the overall Lotus Notes environment, and eliminate its associated software 17

license, server hardware, server software, and storage costs. This project will also eliminate operational 18

issues associated with using obsolete Lotus Notes software and maintaining co-existence of Lotus Notes 19

and Office 365. These 230 Lotus Notes business solutions support OU operations in T&D, Customer 20

70 Refer to WP SCE-04, Vol. 2 Bk A p. 217. 71 Refer to WP SCE-04, Vol. 2 Bk A pp. 218-223.

CIT-00-OP-CS-000067 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total

Recorded / Forecast - - - - - - 3.90 3.00 - - 6.90


Recorded Forecast

47

Service, HR, Legal, IT, Power Supply, Operational Services, Finance, and Corporate Communications. 1

These Lotus Notes solutions will be migrated to existing SCE standard tools such as SAP, Microsoft 2

SharePoint, and BMC Remedy. In doing so, this project will provide for the following operational 3

benefits: 4

1. Simplify SCE’s technology tools by reducing the number of email systems we 5

maintain and support; 6

2. Maximize SCE’s investment in Microsoft Office 365; 7

3. Eliminate the need for a major upgrade to Lotus Notes in 2018 due to technology 8

obsolescence; 9

4. Eliminate operational issues associated with using obsolete Lotus Notes software and 10

maintaining co-existence of Lotus Notes and Office 365; 11

5. Reduce ongoing maintenance costs by decommissioning all remaining components of 12

Lotus Notes at SCE; and 13

6. Improve data governance and data monitoring capabilities through adoption of Office 14

365 functionalities. 15


In 2014, SCE migrated email, desktop and collaboration tools from Lotus Notes to 17

Microsoft Office 365. The key deliverables of the Office 365 project were the migration of 17,500 email 18

boxes, 1,500 document libraries, and the update of the office productivity tools. The successful 19

completion of the Office 365 project in 2015 now enables SCE to simplify our Lotus Notes business 20

solutions by migrating them to Office 365 or decommissioning them. 21

There are 230 complex Lotus Notes business solutions and 202 shared Lotus Notes mail 22

boxes operating at SCE. As part of this project, SCE will migrate approximately 44% of these 23

applications and decommission the remaining 56% of applications. Of the applications to be migrated, 24

40% will be migrated to Microsoft SharePoint, 22% will be migrated to existing business applications 25

such as OpenText or Remedy, 21% will be migrated to third-party or custom applications, and 17% will 26

be migrated to SAP. Migration or decommissioning of these business solutions and shared mail boxes 27

will facilitate savings in hardware refresh, software license, and maintenance/upgrade costs estimated at 28

$9.66 million over 5 years once the project is implemented. 29

48


Project costs include all project management, organizational change management, data 2

migration services, application development services, testing services, and software and hardware 3

components.72 This estimate is based on analysis and estimation of migration costs associated with 4

existing business solutions. In addition, costs for decommissioning of all remaining Lotus Notes 5

components have been included. The project forecast is included in Table IV-13. 6


Alternative: Continue to use Lotus Notes as a business solutions technology. This 8

alternative is not recommended due to the following reasons: 9

1. This alternative would cost SCE an additional $5.64 million over 5 years 10

between 2017 and 2021 to maintain both Lotus Notes and Office 365 email and application services. 11

These costs include an additional $4.960 million for Lotus Notes email software licenses over 5 years 12

between 2017 and 2021 and approximately $0.7 million to upgrade Lotus Notes software, servers, and 13

storage to maintain continued vendor support, operations, and cybersecurity compliance in 2017; 14

2. This alternative would not remediate the operational and usability issues 15

associated with using obsolete Lotus Notes software and maintaining co-existence of Lotus Notes and 16

Office 365; and 17

3. This alternative would prevent SCE from fully realizing the value in our 18

investment and use of Office 365. 19

72 Refer to WP SCE-04, Vol. 2 Bk A p. 223.

49

C. Backup and Disaster Recovery Optimization 1

Table IV-14 Backup and Disaster Recovery Optimization73



Disaster Recovery (DR) includes the strategies, plans, and computing infrastructure to 3

minimize interruption and recover business application systems if a disaster occurs. SCE currently 4

maintains DR capabilities through redundant computing capabilities at SCE’s two data centers, in 5

Alhambra and Irvine, such that when a Mission Critical Application (MCA)74 fails at one location, the 6

failure is mitigated by the resumption of operations by its redundant counterpart in the other location. 7

This project will mitigate deficiencies in the current disaster recovery environment and replace existing 8

backup systems with newer technologies. Improvements in DR will improve SCE’s ability to enable 9

restoration of business functions, associated with mission critical systems, at either data center during a 10

disaster. 11

This project will implement new technology to improve DR capabilities in the following 12

areas: 13

• Install DR hardware and software in each data center to provide adequate support 14

should a critical application need to operate from a non-primary data center location 15

in the event of a disaster. DR hardware and software is composed of data storage, 16

network equipment, and computer processing units. 17

• Conduct controlled DR scenarios in the production environment to provide data 18

replication and system recovery for all required components. 19

73 Refer to WP SCE-04, Vol. 2 Bk A pp. 224-229. 74 SCE’s Mission Critical Applications (MCA) include applications that support power procurement,

transmission and distribution, and customer service business processes, such as the Outage Management System, Energy Management System, Power Costs Inc. (energy trading), and Customer Service System Account Management.

CIT-00-DM-DM-000140 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 2.00 2.30 1.00 0.50 5.80 Previous GRC Request - - - - - - - -

Recorded Forecast

50

• Implement new technical methods in which DR strategies are implemented for 1

applications, leveraging cloud computing concepts. 2


As technology advances in data center management, SCE has identified opportunities to 4

improve and advance the data recovery capabilities in the event of a disaster. SCE’s MCAs currently 5

leverage DR capabilities, but there is a lack of consistency in how each MCA uses technologies to 6

execute DR capabilities. This lack of consistency and standardization creates higher and unnecessary 7

complexity in the maintenance and execution of DR capabilities for these applications, which results in 8

longer recovery times in the event of a disaster, and increased costs to operate the DR solutions relative 9

to the solution this project will provide. Additionally, newer technologies have come to market in recent 10

years that enable increased reliability in methods of data recovery. Failure to remediate and improve DR 11

capabilities of these systems will impact our ability to restore business operations during a disaster. In 12

the event of an outage on a critical application, service to our customers, such as billing operations, 13

would be impacted. 14


This project is scheduled to begin in 2017 and will be completed in 2020. Total project 16

costs are forecast to be $5.8 million. The forecast for this project includes procurement and installation 17

of new hardware along with project team costs for SCE employees, supplemental workers, and 18

consultants to implement and test the upgraded DR environments.75 19

The first phase of this project will install storage, network, and computing resources for 20

DR capabilities associated with MCAs. SCE will transition from traditional magnetic tape backup 21

technology to newer spinning disk and cloud-computing-based approaches. To enable this transition 22

from magnetic tape, new computer processing units (also known as “appliances”) must be installed. In 23

addition to this, disaster recovery software is required to orchestrate the transfer of information and 24

recovery process across various data centers. The end result of this updated design will provide faster 25

data backup and restoration times in the event of a disaster. 26

The second part of this initiative involves migrating the DR capabilities of identified 27

MCAs to leverage the new disaster recovery approach. This transition will be completed in a phased 28

75 Refer to WP SCE-04, Vol. 2 Bk A p. 229.

51

manner based on prioritization of MCAs. The work activities during each migration will require SCE 1

employees, supplemental workers, and consultants to design, build and test the new DR functionality 2

leveraging infrastructure installed during the first phase. 3


Alternative 1: Do not plan for automated disaster recovery capabilities across 5

data centers. In the absence of automated disaster recovery, SCE would rely on responding to a disaster 6

by identifying interim computing resources at an alternative site and restoring data from physical backup 7

media. This approach would add days or weeks to the recovery process and restoration of critical 8

services. If the target computing resources are not available, resources might have to be procured, which 9

can also add to the length of an application recovery. 10

Alternative 2: Plan for all MCA DR to be provided by an offsite service provider 11

and rely on this service provider to provision and test DR functions. This is not currently being 12

considered due to the data vulnerability risk associated with transitioning to a traditional public cloud 13

service. 14

D. Information Technology Projects less than $3 Million 15

Table IV-15 Information Technology Projects less than $3M76

Work Breakdown Structure (WBS) Forecast Capital Expenditures (Nominal $Millions)

Table IV-15 lists capitalized software projects with total project costs less than three million 16

dollars in capital funding and will start and finish within the years 2016 to 2020. Please see the 17

workpapers for these projects for detailed information on each project and its associated costs. 18


WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-DM-DM-000072 Enterprise Schedulers Consolidation - - 2.25 - - 2.25 CIT-00-DM-DM-000073 Database Backup Optimization - 0.40 1.50 - - 1.90 CIT-00-OP-CS-000065 CITRIX VDI Capacity Increase 1.00 - - - - 1.00 CIT-00-DM-DM-000144 User Experience Technologies - 0.50 0.80 1.00 0.30 2.60 CIT-00-DM-DM-000145 Application Distribution - 0.40 1.20 0.80 - 2.40 CIT-00-DM-DM-000146 Modernize Tools for Software Development - 0.50 1.50 - - 2.00

Total 1.00 1.80 7.25 1.80 0.30 12.15

52

V. 1

OPERATING UNIT SOFTWARE PROJECTS 2

A. Customer Service Software Projects 3

This section addresses SCE’s 2016-2020 forecast for capitalized software projects to support the 4

Customer Service OU. This request includes the following four projects, each over $3 million: (1) 5

SCE.com Strategic Upgrade/Stabilization, (2) Digital Customer Self-Service, (3) Alerts & Notifications, 6

and (4) Meter Data Management System (MDMS) Upgrade. As described in SCE-3 and SCE-4, Volume 7

3, a key focus for SCE is modernizing our customer care technology portfolio and systems to deliver 8

simple and efficient solutions. This includes providing greater digital capabilities for customers, 9

managing customers’ contact and alert preferences, and maintaining current software to deliver accurate 10

customer bills. As further described below, the Customer Service capitalized software projects requested 11

in this rate case are proposed in support of meeting these objectives. 12

1. SCE.com Strategic Upgrade/Stabilization 13

Table V-1677 SCE.com Strategic Upgrade/Stabilization


a) Project Description 14

The Commission approved the SCE.com Strategic Upgrade project in SCE’s 2012 15

GRC, and subsequently in SCE’s 2015 GRC.78 While SCE has completed the majority of work 16

approved in previous GRC applications, there is remaining work to be completed in 2016. This 17

77 Refer to WP SCE-04, Vol. 2 Bk B pp. 6-8. 78 Please see D.12-11-051, p. 859, for the authorized forecast costs in 2011 and 2012 with a 10% reduction for

all SAM projects including SCE.com and D.15-11-021 for the authorized forecast for the remaining scope of work for this project.

Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 8.66 13.52 22.64 9.17 9.24 6.15 - - - - 69.38 2015 GRC Authorized* 8.66 13.52 22.64 13.80 - - - 58.62 2015 GRC - Original Request 8.66 13.52 30.63 13.80 - - - 66.61 * In D.15-11-021, the Commission adopted SCE's 2013 recorded costs and 2014 request.

Multiple CITs: CIT-00-DM-DM-000022, CIT-00-DM-DM-000024, CIT-00-OP-CS-000022.

Recorded Forecast

53

testimony updates the Commission on the remaining scope of work to be completed in this rate case 1

period and a summary of the completed scope of work. 2

The SCE.com Strategic Upgrade/Stabilization project replaces existing SCE.com 3

legacy web platform with a modernized IBM WebSphere web platform, and it includes the migration of 4

existing core applications to a new platform. This enables SCE to support the continued use of the 5

website as a robust source of information for rates, programs, and services, while enabling new and 6

contemporary self-service tools for customers to complete core transactions online.79 The project was 7

originally intended to span four years, beginning in 2011 and ending by 2014; but due to program design 8

changes, and longer periods required to test and stabilize functional and technical capabilities, the 9

project completion date has been extended to 2016. 10

b) Need for Project 11

SCE.com’s prior platform and architecture had reliability risks and could not fully 12

support the changing business needs of SCE and its customers. The SCE.com applications lacked a 13

standards-based design, rendering the entire system difficult to maintain and scale. If SCE did not 14

implement this project, extensive reprogramming efforts would have been needed to accommodate 15

increases in the types and volumes of transactions SCE expects in the near future. For these reasons, the 16

Commission approved SCE’s 2012 and 2015 GRC projects to replace the current SCE.com 17

infrastructure. 18

c) Scope 19

(1) Remaining Scope to be Completed in 2016 20

The remaining scope of this project includes migration of the existing My 21

Account and Billing and Payment options to the new platform, which is targeted for completion in 2016. 22

While many of the My Account information items were provided under the prior platform, this project 23

will migrate these same offerings to the new platform so customers can access these features through the 24

79 SCE considers core transactions to include Service Enablement: allowing customers to submit their electric

service turn on / off / transfer requests; Billing: access to current bills and billing history; Payment (credits / arrangements): ability for users to submit payments or set up payment arrangements; Outage: allowing customers to quickly report an outage, and receive updates about outages via the channel of choice (email, text, or voice message); and Usage Data: easy access to energy usage information to help customers understand their bills.

54

updated WebSphere platform. A high-level summary of the completed project scope for migration of 1

core legacy .NET applications is provided below. 2

(2) Completed Scope 3

The following scopes of work have been completed through 2015: 4

Online Turn-On, Turn Off, or Transfer (Move Center): Customers are 5

now able to initiate a turn-on, turn-off, or transfer-service-location request any time online.80 Additional 6

foundational capabilities have also been implemented with the SCE.com Move Center such as 7

“shopping cart” and “check out” functionality for enrolling in SCE Products/Services. 8

Device Flexibility/Responsive Design: By auto-detecting device access 9

format requirements, the webpages resize and present content appropriate for the device. This is known 10

as adopting a mobile-first design approach, which provides a multiplatform foundation that delivers 11

content parity for any device. Additionally, this eliminates the need to create separate websites with 12

content optimized for desktop, tablet, and mobile device access. The remaining programs to be migrated 13

to the mobile-first design approach include My Account and Billing and Payment options. The project 14

scope to complete this work is a part of the Digital Customer Self-Service project. 15

Outage Center: Customers have been provided with a stand-alone mobile-16

first outage center website where customers can report outages and street-light outages through their 17

mobile devices. Through Google Maps, customers have an at-a-glance view of current outages across 18

SCE’s territory with one-click access to additional details on the outage, such as the cause of the outage, 19

the status of the repairs, and the estimated restoration time. 20

Stabilization: Upon implementation of the initial phase in 2013 and 21

subsequent releases of SCE.com on the new IBM WebSphere platform in 2015, a period of stabilization 22

was required to focus on improving site performance. This effort also included identification and 23

resolution of underlying platform issues created by the necessity of having both SCE’s legacy web 24

platform and the new WebSphere platform in co-existence during this transition. The primary objectives 25

during this stabilization period were to reduce the number and severity of Business Impact Events 26

80 Customers who opt out of the Edison SmartConnect® program or non-residential customers can also use the

online process; however, the service transaction does not currently use the Remote Service Switch (RSS).

55

(BIEs);81 address the root cause of recent BIEs; understand and reduce the risk of all deployments that 1

impact SCE.com; and improve the process for future releases. Each stabilization period was a six-month 2

effort requiring all in-progress work to be temporarily placed on hold until the achievement of functional 3

and technical stability. 4

d) Recorded and Forecast Expenditures 5

The capital forecast for this project was developed using SCE’s internal cost 6

estimation model along with the recorded expenditures for 2011-2015. This model utilizes industry best 7

practices and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this 8

project includes costs for labor, hardware, licensing, and other costs. See this project’s workpaper for the 9

cost breakdown information.82 10

SCE forecast $6.15 million in 2016 to complete the remaining scope of work for 11

the SCE.com Strategic Upgrade/Stabilization project. The recorded and forecast project expenditures 12

from 2011-2016 total $69.38 million, which is $2.77 million above our 2015 GRC forecast of $66.61 13

million. The increase in project expenditures is due to the need to resolve technical and stabilization 14

issues beyond the original project scope and forecast that occurred in 2013. These challenges delayed 15

completion of the planned and previously authorized original scope of work from 2013 to 2014 and 16

2015. The difference between the updated total project forecast and the 2015 GRC revised authorized 17

project costs is discussed below. 18

In SCE’s 2015 GRC, SCE forecast project expenditures of $66.61 million to 19

complete the scope of work under a revised timeline. In D.15-11-021, however, the Commission 20

authorized 2013 recorded costs for all capitalized software projects. For the SCE.com Strategic 21

Upgrade/Stabilization Project, SCE spent approximately $8 million less in 2013 than originally forecast. 22

The Commission authorized $58.62 for this project in the 2015 GRC, as shown in Table V-16, which 23

reflected the $8 million underspend in 2013. 24

81 A Business Impact Event (BIE) is an incident that causes disruption to an IT service (e.g., application failure),

which ultimately impacts the business user. BIEs track and monitor the incident through its lifecycle and regularly communicate the status and mitigation plans to SCE business users.


56

2. Digital Customer Self-Service 1

Table V-17 Digital Customer Self-Service83

Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditure(Nominal $Millions)


The purpose of the Digital Customer Self Service (DCSS) project is to expand the 3

customer self-service capabilities of SCE.com by implementing solutions that make it quick and easy for 4

our customers to take action regardless of the device they are using. This project will make possible new 5

electronic billing and payment transactions, will upgrade customer security specifically for program 6

enrollment, and will make website functionality improvements. The project will create O&M savings by 7

streamlining new ebilling enrollments and will result in avoided cost savings related to customer 8

security and authentication. 9

Our customers expect simple, easy, and intuitive interactions with us through a 10

variety of methods, with an increasing reliance on mobile connectivity. Investing in digital as the 11

channel of choice for customer transactions is a primary focus for SCE. SCE aspires to have digital as 12

the channel of choice84 to serve the online needs of our customers. We expect digital core transactions to 13

increase to over 60% by 2020.85 To serve this growth in self-service transactions, SCE is undergoing a 14

major transition in how we view and manage our digital interactions. 15

83 Refer to WP SCE-04, Vol. 2 Bk B pp. 9-21. 84 SCE defines digital or digital as the channel of choice as web and mobile. 85 SCE considers core transactions to include Service Enablement: online ability of customers to submit their

electric service turn on / off / transfer requests; Billing: access to current bills and billing history; Payment (credits / arrangements): ability for users to submit payments or set up payment arrangements; Outage: allowing customers to quickly report an outage, and receive updates about outages via the channel of choice (email, text, or voice message); and Usage Data: easy access to energy usage information to help customers understand their bills.

CIT-00-SD-PM-000237 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total

Recorded / Forecast - - - - - 3.10 7.50 4.00 2.50 1.50 18.60 Previous GRC Request - - - - - - - -

Recorded Forecast

57


Customers are becoming more technologically savvy and expect to be able to 2

complete day-to-day transactions online themselves such as receiving and paying bills, turning service 3

on and off, and accessing news and information.86 Easily transacting with their electricity provider 4

should be no exception. Failed digital transactions result in dissatisfied customers and, more often than 5

not, a higher-cost transaction to SCE.87 6

Part of the Customer Service Strategy is to make it easy for customers to use 7

channels like digital. Enabling and optimizing digital features allow customers to self-serve at their 8

convenience. When a digital transaction is properly designed and reliably built, it will shift and maintain 9

volume to the digital channel, meeting basic customer needs and leading to higher customer 10

satisfaction.88 11

Mobile devices are driving the growth in the digital marketplace, and the 12

transition to mobile is affecting how SCE designs our future digital interactions with our customers and 13

how we provide that service on their preferred channels. Sixty-five percent of smartphone users say 14

paying and receiving bills on a smartphone increases their satisfaction.89 On SCE.com, over 40% of our 15

traffic today is generated by mobile devices and is forecast to grow to 70% by 2020.90 For SCE to meet 16

customer expectations we must develop solutions that work on mobile devices and make completing 17

transactions secure and easy. 18

c) Scope 19

The scope of this project will leverage the enhancements established by the 20

WebSphere Platform discussed in the SCE.com Upgrade project. The project scope includes an 21

integrated set of solutions: Device Support, Electronic Billing and Payment transactions, Security and 22

Authentication, and Website Functionality Improvements. These project scope components are 23

discussed below. 24

86 Refer to WP SCE-04, Vol. 2 Bk B pp. 22-24. 87 Refer to WP SCE-04, Vol. 2C pp. 1-16. 88 Refer to WP SCE-04, Vol. 2 Bk B pp. 23-30. 89 Eric Leiserson, “Mobile Billing and Payment: Consumer Preferences and Billers’ Strategic Response,”

Fiserv, p. 12. 90 Refer to WP SCE-03, Ch. VII-X, p. 148.

58

(1) Device Support 1

The project scope for Device Support provides for the continued 2

implementation of our mobile-first framework, which auto-detects device access format requirements 3

and resizes webpages to present appropriate content, making it quick and easy for our customers to take 4

action regardless of the size of the screen. By allowing customers to use their device of choice when 5

transacting, we will increase customer usage of SCE.com and the volume of self-service transactions. 6

This device-support framework, which starts with responsive design, 91 is applied to each new and 7

enhanced digital feature, including development of conceptual wireframes, clickable prototypes, 8

execution of customer research, user experience implementation, cross-browser and device testing. 9

(2) Electronic Billing and Payment Transactions 10

The project scope includes My Account / Billing and Payment 11

optimization that is focused on improving customers’ ability to easily view and pay bills and on 12

improving the main Billing & Payment webpages. These capabilities will be enabled once the migration 13

to the WebSphere platform is complete. This project will consolidate existing websites, webpages, and 14

mobile applications into one integrated experience to simplify and make consistent the online customer 15

experience. The Electronic Billing & Payment (B&P) scope is a key enabler for increasing Paperless 16

Billing adoption. Increased customer participation in electronic billing is forecast from enrolling 17

customers who choose to pay electronically (e.g., credit card, EFT, SCE ePay) automatically in the 18

Paperless Billing program. 19

The increased customer participation in electronic billing will save 20

postage expense and billing production costs (e.g., paper, ink, printing costs). Additionally, other 21

electronic billing enrollment initiatives including marketing efforts will require SCE.com programming 22

and modifications to achieve the forecast number of new ebilling customer enrollments. The annual 23

forecast for 2018-2020 in O&M reductions for increased customer participation in ebilling is $4.839 24

million for Postage Expense (FERC 903.100) and $1,257 million for billing (903.500).92 The forecast 25

savings in postage expense and billing expenses depend on approval for this capitalized software project 26

or the reductions will need to be removed from the O&M forecast. Table V-18 below shows the forecast 27

91 Responsive web design is a webpage approach aimed at allowing webpages to function on multiple devices,

independent of display device (e.g., desktop monitors, mobile phones, tablets). 92 See SCE-03, Chapter IV. B. for Billing and Chapter IV. D. for Postage testimony.

59

number of new ebilling transactions for 2016-2018 and the 1.25 million in additional customers 1

expected to participate in ebilling by the Test Year. 2

Table V-18 SCE Planned Electronic Billing Program Initiatives

Forecast New Customer Ebilling Transactions and Enrollments 2016-2018

(3) Security and Authentication 3

This project will also enhance our existing Authentication framework 4

while upgrading our customer-facing identity management security. Authentication is one of the largest 5

hurdles for self-service transactions for SCE customers expecting to access and complete core 6

transactions without a lengthy, complex registration and log-in process. The project scope for Security 7

and Authentication includes simplifying account set-up and password resets that allow for guest 8

transactions through Two-Factor authentication.93 Online Two-Factor authentication will increase the 9

security of authentication and increase the success rate of customers signing on. Currently, SCE.com 10

requires an account number to register as a user, which can be inconvenient for customers unable to 11

93 Two-Factor Authentication is a security process in which the user provides two forms of identification from

separate categories of credentials.

Line No.

Description 2016 2017 2018

Estimated New Ebilling Transactions1 Default SCE ePay Customers 2,201,859 1,169,289 - 2 EV Sweepstake - 960,000 480,000 3 Default Credit Card Customers 379,512 212,796 - 4 Default DP Customers 486,918 462,306 - 5 Offer Paperless to remaining - 210,000 126,000 6 iPad Promotion 187,150 88,850 - 7 Marketing 537,207 500,000 8 New Initiatives 997,580 1,045,129 544,871 9 Default EFT Customers 345,000 1,635,000 - 10 Total Yearly New Ebilling Transactions 5,135,226 6,283,370 1,150,871 11 Total Cumulative New Ebilling Transactions 5,135,226 11,418,596 12,569,467 12 Weighted Average Postage Rate $0.385 $0.385 $0.38513 Cumulative Postage Savings $1,977,062 $4,396,159 $4,839,24514 Postage Study Savings in $000 $1,977 $4,396 $4,839

15Estimated Number of New Ebilling Customer Enrollments based on 10.05 mailings per year 510,968 1,136,179 1,250,693

60

quickly find their bill and locate their account number. The project will allow SCE.com users to choose 1

from multiple options to authenticate (e.g., phone number, zip code, address, and social sign-on). This is 2

expected to avoid increasing the number of calls placed to our Customer Call Center (CCC) to assist 3

with these requests. Additionally, the project scope includes providing different levels of account access 4

through roles/permissions to further the customer experience and protect the account holder from 5

unauthorized changes to their SCE account. 6

Table V-19 below provides the forecast avoided cost savings from the new 7

Authentication framework. The forecast is based on estimated number of calls to the CCC by transaction 8

type, historical percentage of customers who failed to complete authentication and then exited SCE.com 9

or clicked on “Contact Us,” and forecast increase of customers using SCE.com post-authentication 10

improvements. The forecast avoided cost savings to be realized with implementing the Authentication 11

initiative is forecast to be $3.5 million over 2016-2020 period.94 12

Table V-19 Digital Customer Self-Service Project

Authentication Framework Forecast Avoided Costs 2016-2020

(4) Website Functionality 13

The Project Scope for Website Functionality centers on targeted SCE.com 14

user experiences and capability improvements by using data-driven customer research, key performance 15


Line No. Description 2016 2017 2018 2019 2020 Total

1Projected Increased SCE.com Transaction Growth 48% 54% 59% 63% 66% n/a

2 Password Reset 58$ 246$ 259$ 268$ 278$ 1,109$ 3 User ID Recovery 10$ 42$ 44$ 46$ 48$ 190$ 4 My Account Registration 325$ 456$ 472$ 488$ 1,741$ 5 Summer Discount Program 12$ 26$ 27$ 28$ 93$ 6 CARE Enrollments 43$ 91$ 94$ 97$ 325$ 7 Other Program Enrollments 3$ 6$ 6$ 6$ 21$

8Total avoided cost of customer calls related to authentication 68$ 671$ 882$ 913$ 945$ 3,479$

61

indicators, and metrics. Web intelligence is a combination of digital and business metrics that examines 1

the performance of a website and visitor’s interactions with site features. SCE can then measure the 2

impact of these interactions across the enterprise. SCE’s ability to leverage improved web intelligence to 3

make customer-focused, data-driven decisions along with customer testing and internal performance 4

testing will be a key shift in how SCE sets its website priorities. Improved website data and metrics 5

coupled with knowledge from visitor interactions will allow us to present and offer the right site features 6

and content, and measure the impact of these interactions across the enterprise. The following are 7

examples of website functionality improvements: 8

• Optimizing the Customer Support pages based on the results from data 9

driven metrics and analytics; 10

• Evaluating the “Pay as Guest” vs. “Log In & Pay” changes from the 11

previous year to further optimize to drive customer satisfaction; 12

• Strengthening metrics to allow us to better pinpoint user experience 13

problems and improve customer satisfaction ratings and increase 14

online adoption; 15

• Updating the Billing & Payment pages to promote Paperless Billing 16

and streamline content so it is easier to identify additional options; and 17

• Updating the Home Page, Move Center, and Outage Center pages to 18

promote key features customers care about most and streamline 19

content so it is easier to identify additional options. 20

Along with Web Intelligence functionality, digital support services can 21

assist us in retaining customers in the digital channel. Digital support services include the integration of 22

intelligent virtual assistance for customers with a constantly improving knowledge base, resulting in 23

seamless interactions that increase customer satisfaction and conversion rates. Forrester states: 24

62

“Web chat will grow because it preserves web self-service context. 1

Online chat adoption among US online adults has significantly risen in 2

the last five years — from 38 percent in 2009 to 43 percent in 2012 to 3

58 percent in 2014. Chat offers several benefits to the customer: Firms 4

can quickly connect customers to an agent with the right skills to 5

answer the question without having to navigate an arduous interactive 6

voice response (IVR); questions can be succinctly resolved in near 7

real-time; and agents can leverage customer behavior on the website 8

to move the conversation forward instead of rediscovering information 9

that has already been communicated to the customer.” 95 10

d) Forecast Expenditures and Cost-Benefit Analysis 11

The Digital Customer Self-Service project is a five-year project beginning in 2016 12

and is planned to be completed in phases through 2020 with a forecast total expenditures of $18.6 13

million.96 The capital forecast for this project was developed using SCE’s internal cost-estimation 14

model. This model utilizes industry best practices and SCE subject matter expertise to estimate project 15

cost components. SCE’s forecast for this project includes costs for labor, hardware, licensing, and other 16

costs. See this project’s workpaper for the cost breakdown information.97 17

The Digital Customer Self-Service project operational benefits result in a benefit-18

to-cost ratio of 1.21.98 As discussed above, Customer Service operational savings relate to ebilling 19

customer enrollments that reduce both postage and billing expenses. IT operational savings of 20

approximately $1.155 million will be achieved by decommissioning servers and lowering licensing costs 21

once the Authentication framework is placed into service. SCE expects to incur project-related expenses 22

that will not be capitalized. These expenses include IT O&M expenses of $1.1 million for vendor 23

authentication software licenses and other IT support and maintenance expenses as well as one-time 24

marketing costs of $750,000. The marketing costs will be incurred to increase customer awareness of the 25

new functionality to conduct core transactions through SCE.com. 26

95 Refer to WP SCE-04, Vol. 2C pp. 17-30. 96 Refer to WP SCE-04, Vol. 2 Bk B p. 21. 97 Id. 98 Refer to WP SCE-04, Vol. 2 Bk B pp. 18-20.

63

The benefit-to-cost ratio analysis assumed an overall project life through 2022 1

(five years after 2017). This is appropriate when the majority of the capital expenditures are in service. 2

The benefit-to-cost analysis also escalated all future costs and benefits to year-of-expenditure, then 3

discounted to 2015 NPV using a 10% discount rate. Finally, the sum of the present-value benefits was 4

divided by the sum of the present-value costs to derive the benefit-to-cost ratio for the project. The 5

benefits and ongoing costs from this project that occur from 2021-2022 will be reflected in SCE’s next 6

2021 GRC Test Year. Table V-20 shows the forecast project capital expenditures and O&M savings 7

from 2016 through 2020. 8

Table V-20 Digital Customer Self-Service Project

Benefit to Cost Ratio ($ in Millions)

e) Alternatives Considered 9

The alternative considered for this project was to maintain the current state of 10

SCE.com without the necessary improvements, increased functionality, and O&M savings that this 11

project is designed to deliver through the IBM WebSphere Portal framework. The result of maintaining 12

the current state of SCE.com without this project would be removal of the forecast benefits of over $25 13

million from 2016 through 2020 for new ebilling enrollments. Additionally, the avoided cost of $3.5 14

million would need to be added to the O&M forecast as an increased number of customer calls would 15

occur without the improved security and authentication that this project will put in place. Lastly, 16

increased customer usage of SCE.com as a key tool to improve customers’ management of energy 17

consumption and participation in utility programs is less likely to be achieved. 18

Revenue RequirementNominal Present Value

1 Project Capital 18.60$ 2 Total O&M 9.37$ 3 Total Costs 27.97$ 26.21$ 4 Benefits 43.51$ 31.84$ 5 Benefit to Cost Ratio 1.21

Line No.

Description

64

3. Alerts & Notifications 1

Table V-21 Alerts & Notifications99


The Alerts and Notifications project will provide timely and accurate digital 2

communications to customers, maintain customer data privacy, and facilitate compliance with the 3

Telephone Consumer Protection Act (TCPA) and Controlling the Assault of Non-Solicited Pornography 4

and Marketing Act (CAN-SPAM) federal laws. This project is the consolidation of two capitalized 5

software projects that were authorized in our 2015 GRC: the Alerts & Notification project and Outage 6

Communication project.100 This testimony will update the Commission on the completed scope of work, 7

remaining scope of work, and recorded and forecast costs to complete the project. 8


The Alerts & Notification project will provide a new centralized and flexible 10

platform of systems to store customer contact information and alert preferences, and tools to manage 11

those preferences. The new platform will support, among other things, the following system capabilities: 12

(1) centralized storage of all customer contact information and digital alert preferences, (2) functionality 13

to auto-enroll customer contacts into various alerting services (e.g., outage alerts), (3) customer ability to 14

directly manage their contact information and alert preferences (e.g., outage, demand response, billing 15

and payment) using self-service tools (e.g., web preference center and Interactive Voice Response 16

(IVR)), (4) delivery of alerts and notifications including the packaging, scheduling, and messaging 17

99 Refer to WP SCE-04, Vol. 2 Bk B pp. 31-36. 100 See D.15-11-021, pp. 249-254.

Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - 0.50 7.64 0.60 4.30 4.90 0.60 - 18.54 Previous GRC Request* - - - 3.40 12.30 4.10 - 19.80 *The Commission adopted SCE's request for this project in D.15-11-021.

Multiple CITs: CIT-00-SD-PM-000107, CIT-00-SD-PM-000238, and CIT-00-SD-PM-000171

Recorded Forecast

65

alignment to enrolled customers, and (5) improved reporting on customer contact enrollment, customer 1

type, alert preferences, and unenrollment reasons. 2


In 2014 and throughout much of 2015, SCE used multiple isolated systems to 4

store contact and alert preferences, and multiple notification systems to send alerts to customers. The 5

new platform will consolidate multiple core and auxiliary support systems into two systems: a contact 6

and preference management system (single source of accurate information) and a notification system. 7

The new platform reduces the complexity of SCE’s technology portfolio. It also increases customer 8

adoption by allowing customers to enroll and receive timely and accurate alerts for the programs and 9

services they choose, and to select the digital channel they prefer most (email, voice or text).101 Digital 10

alerts will include core electricity service alerts such as outage alerts and final call notices, and optional 11

alerts and notification such as bill ready notification, demand response program events, and energy 12

efficiency recommendations.102 13

The new Alerts & Notifications platform is necessary to provide integrated and 14

scalable systems to support compliance with TCPA and the CAN-SPAM federal laws. The new platform 15

will store customer consent to use their landline, mobile phone, or email address for automated digital 16

communications, which will reduce SCE’s risk of TCPA and CAN-SPAM non-compliance. This portion 17

of the project scope will develop and maintain a centralized repository to capture customer “Do Not 18

Contact” preferences for meeting the “Do Not Call” requirements set forth in various federal statutes.103 19

c) Completed Project Scope 20

SCE is using a phased development approach for the Alerts and Notifications 21

project and has implemented Releases 1 and 2a. These two releases are discussed below: 22

Project Release 1 – Implemented in September 2015 23

Release 1 provided the capability for approximately 300,000 eligible small 24

business customers to optionally enroll and receive proactive maintenance and repair outage alerts via 25

101 Refer to WP SCE-04, Vol. 2 Bk B pp. 35-36. 102 There are approximately 3.5 million maintenance or repair outages each year that impact residential and non-

residential customers. 103 E.g., Mobile Marketing association and Direct Marketing Association Code of Conduct, California’s Shine

the Light Law, FCC Telephone Consumer Protection Act, Telemarketing and Consumer Fraud and Abuse Prevention Act, CAN-SPAM.

66

their channel of choice (email, voice, or text message). This release improved and simplified outage alert 1

message content and format for all enrolled customer types and facilitated self-service opt-out of outage 2

alerts when desired. The opt-out process also identifies when an outage alert delivery failed due to 3

inactive phones and email addresses, which enables SCE to discontinue sending unnecessary outage 4

alerts. The total cost to implement Release 1 was $1.3 million. 5

Prior to Release 1, small business customers only received initial maintenance 6

outage alerts via U.S. Mail or door hangers. When there were outage schedule changes a few days 7

before the outage there was rarely enough time to inform impacted customers before the original outage 8

date. Release 1 addressed this issue by enabling digital maintenance outage updates throughout the 9

lifecycle of the outage so small business customers receive sufficient time to prepare and help ensure 10

their employees and customers remain informed. 11

Project Release 2a – Implemented in December 2015 12

Release 2a established a new SAP CRM platform for outage alert preferences that 13

stores and manages small business and residential customer contact information and digital outage alert 14

preferences. The new SAP CRM platform delivers the following functionality: (1) enabled residential 15

customers to enroll and receive proactive maintenance and repair outage alerts using self-service tools, 16

(2) enabled new customer self-service tools to manage their outage alert preferences on SCE.com, (3) 17

established auto-enrollment for eligible customer contacts in outage alerts, (4) automated the processing 18

of outage alert customer opt-outs and unenrollment for inactive phones and email addresses from 19

required systems, and (5) provided new and improved outage and alert enrollment reporting. Release 2a 20

established the base framework of the new platform that will help SCE create a simple and efficient 21

outage experience for our customers. For Release 2a, the recorded costs to implement this project were 22

$6.8 million in 2015. 23

d) Remaining Project Scope 24

Project Release 2b – Planned Implementation in Early 2017 25

The next phase of this project is Release 2b, which will refine our customer 26

contact and outage alert preference self-service tools and the actual content of the outage alerts based on 27

customer feedback. Release 2b scope includes (1) simplifying the customer self-service alert-preference 28

management online experience by making it easier to enroll in outage alerts, (2) introducing new 29

customer self-service alert preference management tools using Interactive Voice Response (IVR), and 30

67

(3) refining outage alert preparation and send rules to improve the timeliness and accuracy of outage 1

alerts sent to enrolled contacts. 2

Release 2b is expected to improve our customers’ outage experience satisfaction. 3

This Release will also help to avoid increasing customer calls to the Customer Contact Center (CCC) to 4

update their contact information and outage alert preferences, by providing customers with effective and 5

easy-to-use self-service tools along with timely and accurate outage alert communications. 6

Future Project Releases – Planned Implementation in 2018 and 2019 7

Additional releases are needed to continue expanding and enhancing SCE’s 8

digital Alerts and Notifications platform to realize all the business capabilities described above. The 9

remaining scope includes the following: 10

• Migrating contact and alert preference information from multiple systems 11

(i.e., legacy Customer Service System (CSS) and SCE.com) into the new 12

platform; 13

• Decommissioning of multiple legacy and aging contact management and 14

notification systems where possible; 15

• Providing new customer self-service and employee tools, such as: 16

o Customer contact information management (e.g., name, phone, phone 17

type, email address); 18

o Enrollment and preference management for migrated alert programs (e.g., 19

Critical Peak Pricing, Summer Discount Plan, Budget Assistant); 20

o Enrollment in new alert programs; 21

o Preference management for core utility alerts (e.g., final call, pending 22

discount, rotating outages); 23

o Other communication preferences management (e.g., marketing 24

communication opt-in/opt-out, Do Not Contact, and mobile consent); 25

• Enabling lower cost digital alert capabilities for migrated and new alert 26

programs; and 27

• Expanding and improving contact information, digital alert participation, and 28

alert reporting. 29

68

e) Recorded and Forecast Expenditures 1

As discussed above, the total cost to implement Release 1 was $1.3 million 2

($500,000 in 2014 and $800,000 in 2015). These costs included SCE and vendor labor costs to support 3

the design, construction, testing, implementation, and warranty support. For Release 2a, the recorded 4

costs to implement this project were $6.8 million in 2015, which included costs for SCE and vendor 5

labor to support the design, construction, testing, and implementation, as well as for new hardware and 6

software licenses. The overall forecast for the Alerts & Notifications program is $18.54 million, which 7

includes the recorded costs for Release 1 and 2a and forecast expenditures for Release 2b and future 8

releases from 2016 through 2019.104 9

4. Meter Data Management System (MDMS) Upgrade 10

Table V-22 Meter Data Management System Upgrade105



In SCE’s 2015 GRC, the Commission authorized upgrades to the Meter Data 12

Management System (MDMS) and the Network Management System (NMS) as part of the Enhanced 13

Metering and Usage project. SCE completed the upgrade of the NMS in 2014. This testimony explains 14

why SCE deferred implementing the MDMS upgrade from 2016 to 2017 and describes the current 15

schedule and forecast for implementation. 16

The MDMS is the repository for the meter and event data collected from the 17

Edison SmartConnect® meters, which is used to bill over five million customers. Because Itron, SCE’s 18

vendor, committed to support the existing MDMS software through 2016, the planned MDMS upgrade 19

104 Refer to WP SCE-04, Vol. 2 Bk B p. 34. 105 Refer to WP SCE-04, Vol. 2 Bk B pp. 37-43.

CIT-00-SD-PM-000236 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - 5.43 1.40 6.70 - - - 13.53 Previous GRC Request* - - - 6.70 - 9.90 - 16.60 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

69

did not needed to be implemented in 2016 as previously anticipated. In 2013, Itron released its latest 1

version of Itron Enterprise Edition (IEE) MDMS software, which results in SCE being one version 2

behind the current product release. If SCE falls behind over two product releases, SCE may lose Itron 3

product maintenance support. Upgrading to the latest Itron IEE MDMS version will also improve 4

functionality and speed of data collection used for billing. To maintain compatibility with the new 5

product release, the new MDMS software requires updating Windows 2003 and Oracle 10.2 software. 6


Stabilizing and maintaining the MDMS environment by implementing the Oracle 8

EXADATA106 solution will minimize unbilled revenues and reduce customer complaints regarding late 9

billing. Further, this project will improve MDMS system reliability such that the Edison SmartConnect® 10

infrastructure can minimize the volume of billing disruptions and reduce customer impacts. The reduced 11

processing time reduces the risk of billing delays, minimizing impacts and complaints from customers 12

who may also rely on the availability of daily data on SCE.com. 13

Edison SmartConnect® relies on COTS products from SCE’s vendor for meter 14

usage data collection from the NMS and the MDMS. To keep the software up-to-date for support, 15

regular vendor upgrades are required (18-24 month cycle). The new enhancement packages are available 16

through SCE’s licensing agreements with the vendor. These enhancements must be made to maintain the 17

reliability of the meter data collection system. 18

The daily collection of meter data is used for bill preparation and made available 19

daily on SCE.com for use by customers, such as our customers participating in Net Energy Metering 20

(NEM) and demand response, who view and track their daily energy use. Also, customers who have 21

enrolled in online account management can view their daily usage on SCE.com. Failures in the data 22

collection process can cause processing delays beyond 24 hours, impacting the following day’s data-23

collection process. These failures can result in late billing and delays in the upload of daily usage data 24

for viewing on SCE.com. The MDMS Stabilization Plan will reduce the daily processing time from 20-25

24 hours to 12-14 hours, and improve the recovery time from system failures. With a shortened 26

processing time, unexpected failures could be resolved within 24 hours and avoid impacting the next 27

106 Oracle Exadata Database is a combined computing and storage system that runs Database software.

70

data-collection cycle. Implementing the upgrade to EXADATA solution will improve the MDMS 1

stability, allowing for faster data loading and recovery times. 2

Besides the Oracle Exadata implementation needed to stabilize the MDMS, the 3

project includes Windows and Itron software upgrades, which are needed for compatibility to support 4

the MDMS. This will also extend product maintenance support by the vendor. If SCE falls behind over 5

two product releases, SCE may lose Itron product maintenance support affecting the end-to-end 6

reliability of the application, affecting the downstream billing processes. SCE resources are not trained 7

to support and maintain proprietary Itron products, so retaining Itron product maintenance support is 8

essential. 9

c) Scope and Cost Forecast 10

The project will be implemented in two phases. Phase 1 includes stabilization of 11

the current MDMS environment and will be completed in 2016. Phase 2 will include software upgrades 12

to preserve vendor support of the system and devices and will be implemented in 2017. 13

(1) Phase 1 14

Phase 1 consists of a hardware upgrade to the Oracle Exadata appliance. 15

This includes replacing the existing IBM AIX hardware with an EXADATA platform. This will upgrade 16

the Oracle version for better management of Oracle processes and improve stability and reliability of the 17

system improving throughput within the MDMS database. The upgrade will improve the data load and 18

batch processes to reduce customer impacts and prevent billing delays. The project focuses on Customer 19

Service databases,107 which will be migrated through the MDMS Stabilization project. 20

(2) Phase 2 21

Phase 2 consists of a Windows upgrade and the IEE 8.x software upgrade 22

to the latest version of IEE, making it compatible with the upgraded versions of Windows and Oracle as 23

required to maintain vendor support. Edison SmartConnect® relies on COTS products from the vendor 24

for meter data collection using the MDMS for managing metering data. The vendor’s products are 25

critical infrastructure components for the SCE-integrated systems, and SCE has a contractual obligation 26

to maintain the software within the latest two release versions to preserve vendor support of the system 27

and devices. These commercial products are enhanced and upgraded by the vendor regularly. Because 28

107 Network Management System (NMS), Cell Relay Configuration Management System (CGS), SCE.com, and

Event Notification System.

71

the system and the process are new, technology and COTS enhancements will occur more frequently 1

than standard. Once the system is in steady state, the duration between upgrades will become longer. 2

This is a common way for software vendors to upgrade their systems as new offerings are developed, 3

improvements in the speed and efficiency of managing data are developed, and problems identified 4

since the previous upgrade are fixed. The new enhancement packages are available to SCE via its 5

licensing agreements with the vendor. 6

d) Recorded and Forecast Expenditures 7

SCE forecasts $1.40 million in 2016 and $6.70 million in 2017 to complete the 8

remaining scope of work for the MDMS project. These expenditures are composed of labor, systems 9

hardware, and licensing costs. The forecast is based upon previous experience with similar COTS 10

projects and work performed on MDMS, labor rates, and cost components. Table V-22 above illustrates 11

the forecast expenditures of the MDMS Stabilization and Upgrade project by year. The overall project 12

forecast of $13.53 million is $3.07 million below the original request of $16.60 million.108 The 13

decreased forecast expenditures are a result of more refined and increased confidence level in the 14

estimates based on more recent information. Furthermore, the product version to be implemented 15

includes SCE requirements in the base software and hardware that will reduce the overall cost. 16

108 Refer to WP SCE-04, Vol. 2 Bk B p. 43.

72

5. Customer Service Projects Less than $3M 1

Table V-23 Customer Service Projects less than $3M109


Table V-23 above lists other capitalized software projects for Customer Service with total 2

project costs less than three million dollars in capital funding and will start and finish within the years 3

2016 to 2020. Please see the workpapers for these projects for detailed information on each project and 4

its associated costs.110 5

B. Transmission & Distribution Software Projects 6

1. Work Management Solutions 7

As part of SCE’s Operational Excellence (OpX) efforts to reduce operating expenses and 8

increase productivity, SCE’s Transmission & Distribution Operating Unit (T&D) identified Work 9

Management process improvement opportunities across key Work Management areas of the Initiate, 10

Plan, Schedule, Execute and Close (IPSEC) process.111 While T&D plans to improve performance with 11

process changes only, technology solution investments are required to fully achieve all Work 12

Management process efficiencies. This section, and the projects that follow, extend the T&D Work 13

Management OpX efforts. 14

T&D’s investment in maintaining and improving the grid has grown over the last 3 years. 15

Managing such a large work portfolio has depended on employees such as planners, schedulers, project 16

managers, analysts, and managers to closely monitor workloads, track task completion, and balance 17

109 Refer to WP SCE-04, Vol. 2 Bk B pp. 44-73. 110 Id. 111 SCE-02, Volume 1, Operational and Service Excellence, Work Management.

WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-SD-PM-000170 2015 GRC Rate Changes 1.10 - - - - 1.10 CIT-00-SD-PM-000239 2018 GRC Rate Changes - 1.00 1.00 - - 2.00 CIT-00-SD-PM-000245 Contact Center Optimization - - 2.90 - - 2.90 CIT-00-DM-DM-000060 Itron 3.9 NMS Upgrade project - - - 1.00 - 1.00 CIT-00-SD-PM-000172 SmartConnect Monitor&Analysis (SCMAS) - - 1.00 - - 1.00

Total 1.10 1.00 4.90 1.00 - 8.00

73

work against resources. Employees across departments and business functions perform these work 1

management activities using independent processes and tools. They do so leveraging a blend of core IT 2

systems (i.e., SAP and Design Manager), outdated Project Management solutions, and standard 3

Microsoft applications such as Excel and Project. Processes, where similar, do not allow for data across 4

departments to be easily integrated. Processes for T&D internal and contract resources are dissimilar, 5

making data sharing and data integration virtually impossible. 6

Work Management within T&D can be improved by optimizing work schedules and 7

resource assignments across the IPSEC process. To meet the increasing demands of the grid and our 8

customers, SCE performed two process assessments with separate independent partners. These 9

assessments resulted in recommendations for improving the way SCE manages its work across the 10

IPSEC model. 11

The “Work Management Assessment” looked at organizations, people, processes, and 12

technology to address key objectives. The objectives include: (1) Increase the accuracy of project scope 13

estimating, (2) Reduce the technology footprint and redundancies by using common tools, (3) Improve 14

and standardize IPSEC workflow processes, and (4) Improve performance through integrating enterprise 15

solutions across the company. The Project Controls Improvement Initiative (PCII) performed a gap 16

analysis between SCE and industry best practices in the areas of project scope, costs, and expenditure 17

controls.112 This initiative identified gaps in the current scope development, cost estimating tools and 18

processes, while also noting the benefits of integrating the scope estimating process with our current 19

enterprise system (SAP). 20

Based on these assessments the scope definitions for each of the IT capital projects 21

(supporting T&D Work Management) were refined to align the business capabilities being sought with 22

the appropriate technical solution. Portfolio management needs originally authorized in the 2015 GRC 23

as the “Integrated Portfolio Management for MPO” were identified to extend beyond MPO.113 This 24

project scope expanded to include project management capabilities. This change was made based on 25

project management capabilities being more aligned with portfolio management than with work 26

management dashboards. The Work Management Dashboard project was reduced accordingly. Another 27

112 Refer to WP SCE-04, Vol. 2 Bk B pp. 77-101. 113 Major Projects Organization (MPO) integration details can be found in the assessments further described in

the subsequent Work Management projects.

74

example of how project scopes were redefined is the Scope Cost Management Tool (SCMT). Project 1

forecasting capabilities were originally planned to be delivered through the SCMT application. The 2

assessments determined that project forecasting capabilities were better suited to be delivered with 3

portfolio management than with SCMT. The scope and costs for these projects were then revised to 4

reflect this change. The scope refinement outcomes are included in the project descriptions that follow. 5

The assessments resulted in a proposed system architecture provided in Figure V-6 below. 6

The series of projects based on the assessments and that are included within the suite of 7

Work Management Solutions include: 8

1. Portfolio Management; 9

2. Scope Cost Management Tool; 10

3. Work Management Dashboard; 11

4. Transmission Telecom Work Order Lifecycle; and 12

5. Click Schedule Refresh Release 1 & 2. 13

These projects will deliver to T&D the increased work management capabilities in 14

portfolio management, resource and budget forecasting, project scope estimating and scheduling, and 15

reporting and analytics. By leveraging standard industry work management tools in a consistent manner, 16

the improved work management capabilities also enable T&D to integrate oversight of work assigned to 17

contract resources. 18

75

Figure V-6 Work Management Assessment

1

2. WM - Portfolio Management 2

Table V-24 WM - Portfolio Management114



CIT-00-DM-DM-000077 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 1.80 6.00 6.20 - - 14.00 Previous GRC Request* - - - 0.40 2.90 1.30 - 4.60 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

76


D.15-11-021 adopted the project as part of SCE’s 2015 GRC, where it was 2

included as part of the “Integrated Portfolio Management for MPO” project. The project was delayed to 3

complete the Work Management Assessment (WMA) described above. Besides delaying the start of this 4

project until late 2016, the WMA confirmed the gaps and benefits were not isolated to “Integrated 5

Portfolio Management for MPO” but all Work Management projects across T&D. This broadened the 6

effort to include all T&D capital projects. 7

T&D capital projects are planned and managed as a portfolio. Each portfolio 8

component represents a collection of projects similar in nature. The similarity is usually based on the 9

asset being worked on, or the construction being performed. For example, substation circuit breaker 10

infrastructure replacement projects follow a common process, similar project timeline, and require 11

essentially the same project management, engineering, electrician, and field construction skillsets. 12

Likewise, transmission reconductor projects have commonalities when compared to each other. 13

Resource requirements and project schedules for transmission reconductor work would not be the same 14

as those for substation circuit breaker infrastructure replacement work. The same analogy would be true 15

for Distribution overhead conductor programs, and all other portfolio components. 16

Conflicts within different projects across the portfolio can occur due to resource, 17

scheduling, or operating constraints. It may not be feasible to perform Distribution overhead conductor 18

replacement concurrent with Transmission reconductoring on the same overhead pole line, from a field 19

coordination perspective. Nor may it be feasible from a grid operations perspective to perform 20

Transmission reconductoring concurrently with Substation Infrastructure Replacement, if the outages 21

required to support each activity cannot be simultaneously managed. Potential resource constraints may 22

also exist while attempting to simultaneously execute Distribution 4kV substation elimination work and 23

Substation Infrastructure Replacement work if both projects depend on the same substation apparatus 24

personnel. The Portfolio Management solution identifies these potential conflicts early to resolve the 25

operational impacts and maximize productivity. 26

By building a long-term plan for the portfolio based on standardized project 27

schedules, the Portfolio Management solution will allow T&D to quantify demands across T&D by 28

work group, asset, circuit or system, and geography. This will allow SCE to assess the organizational, 29

customer and financial implications related to “what if” scenarios or changes in investment plans. If 30

several projects within the portfolio were tracking under budget or behind schedule, the Portfolio 31

77

Management solution would allow the organization to assess various mitigation alternatives. By 1

managing all large projects as a portfolio, schedule or cost deviations of individual projects from their 2

respective project plans can be balanced by adjusting plans for other projects if feasible. 3

Project schedules will be developed based on the standard project template for 4

each area of the portfolio. This will allow SCE to monitor actual project performance against the 5

standard portfolio schedule template and will improve visibility to project risk. Actual project 6

performance, across many projects over time, can be aggregated for each area of the portfolio. The 7

aggregated data can be fed back to improve the corresponding standard portfolio schedule template. This 8

performance-based feedback loop will drive continuous improvement of the portfolio standard schedule 9

and maximize long-term planning efforts. 10

The capabilities delivered would be focused on large capital projects that typically 11

last from a few months to multiple years. The final solution is not designed to address short-duration 12

work (typically from a few days to a few weeks). 13


The level of infrastructure replacement and modernization that SCE expects to 15

execute within the next few years requires effective and efficient long-term planning and project 16

management. Ineffective planning and project scheduling can cause project delays, project over runs, 17

write offs, or frequent cycles of work ramp-up and ramp-down. These results can cascade into 18

overloaded workloads for employees as work spikes and deadlines accelerate. 19

Portfolio and project management capabilities within T&D are performed to 20

various degrees of maturity and across a variety of applications. Substation projects are submitted for 21

portfolio review within a custom-built solution and managed using several older versions of a 22

scheduling tool, which cannot work together. Conversely, Distribution employees rely on Excel and 23

Access to perform reviews of the portfolio. Distribution capital projects are managed using a 24

combination of multiple tools (e.g., SAP, Design Manager, Access, and Excel). The MS Office-based 25

tools are not integrated with SCE’s enterprise applications. Therefore, data for reporting within 26

Distribution are not integrated. Integrating project information between Transmission, Substation, and 27

Distribution is even more problematic. This lack of integration creates an organizational blind spot for 28

resource managers and employees within Substation and Operations. These groups regularly support 29

work plans being driven by Transmission and Distribution projects. Transmission reconductor and 30

distribution 4kV circuit elimination projects may depend on support from grid operators and substation 31

78

apparatus employees. However, the resource managers for these employees do not have access to the 1

project plans or visibility to the timing for the planned work. Instead, they rely on email communications 2

and Excel files for this information. This information can be unreliable and becomes difficult to manage 3

when project dates change. 4

The Portfolio Management project will enable resource managers to balance 5

demand and capacity more efficiently by providing a long-range overview of project needs. This will 6

drive faster project execution. Roles and hand-offs will become more defined and standardized across 7

the system. Implementing Portfolio Management will provide focus on critical activities. It will also 8

minimize rework and false starts resulting from a lack of overall job coordination. The tool will allow 9

resource constraints to be identified earlier to better assess the ramp-up needs of resources (internal or 10

contract) or the resequencing of work, before capital charges record. 11

A common project-management solution that all employees utilize will provide 12

portfolio and project managers a comprehensive view of project status and risk. This will cause drive 13

project delivery or reduced project costs as project risks are mitigated in advance. By meeting these 14

objectives, SCE will improve its overall ability to deliver capital projects as measured by quicker project 15

execution, reduced project delays, timely and effective scope on-ramp/off-ramp, and improved resource 16

and contractor management. Project plan progress can be consistently and accurately monitored, and 17

project plan risk can be mitigated. 18

c) Scope and Forecast 19

The Portfolio Management project will implement a COTS Project and Portfolio 20

Management Solution (PPM) integrated with the balance of T&D’s existing work management 21

solutions. Project scope includes: 22

1. Portfolio Planning & Forecasting—Provide a centralized tool for all large 23

capital programs where program scope, costs,115 resources and schedules are 24

prioritized and approved, as the baseline portfolio investment plan. Provide a 25

tool where project data related to schedule, costs and resources can be 26

aggregated and summarized across capital programs, resource pools and cost 27

categories. Provide the ability to generate forecasts (schedule, budget, and 28

115 Portfolio and project costs would be calculated within the application provided as part of the SCMT project

and input as a project cost estimate by the portfolio or project manager.

79

resource) for capital projects, the programs the capital projects belong to, and 1

the portfolio of capital programs. 2

2. Project Scheduling—Provide a standard tool for all large capital work to be 3

planned and managed allowing for consistent management of schedules and 4

resources and project monitoring against portfolio baselines. Integration of 5

project schedule data into Portfolio Planning solution for risk management, 6

change control governance, and project oversight. 7

3. SAP Integration—Align project schedule structure with recorded costs within 8

SAP work order operations. Provide integration to allow cost data to be 9

provided to project and portfolio managers. 10

4. Contractor Enablement—Provide a tool where project data for project work 11

assigned to contractors can be integrated into the view of the overall portfolio, 12

reducing the specialized hand-offs between groups and eliminating the 13

translation of contractor data into project status views on a delayed basis. 14

Provide a means for contractor collaboration through a standard project 15

documentation exchange and project documentation retention solution. 16

5. Reporting Data—Provide portfolio and project data to enterprise reporting 17

databases. This would be for integration into the centralized reporting and 18

analytics solution with data for work outside the Portfolio Management 19

solution.116 20

The total project cost forecast is $14 million.117 The capital forecast for this 21

project was developed using SCE’s internal cost estimation model. This model utilizes industry best 22


project includes costs for SCE employees, supplemental workers and consultants, software and vendor 24

costs, and hardware costs. See this project’s workpaper for the cost breakdown information. This 25

116 Examples of work not included with the Portfolio Management solution include Distribution Inspection,

O&M and New Service Connections. The reporting needs from this integrated reporting database would be the scope of the Work Management Dashboard project.


80

includes the implementation of COTS Portfolio Management and Scheduling solutions across T&D and 1

integration of these solutions with our SAP enterprise system. 2

(1) Alternatives Considered 3

Alternative 1: SCE considered replacing the existing solution by building 4

a customized solution using SCE and consultant resources or contracting with a third party to build it. 5

We did not pursue this option because it would require SCE and contracted resources to identify the 6

required skills and to set up the development and testing environments. This alternative was also not 7

pursued because it would not be cost-effective to create such a solution because it would require 8

significant future maintenance costs. 9

Alternative 2: SCE considered replacing the existing solution by 10

procuring a new commercially available tool or service. We did not pursue this alternative because it 11

would require SCE to go through the competitive bid process for both the tool and additional support 12

services to integrate it with the existing tools in our IPSEC model. This option was also not pursued 13

because implementing a new tool would also carry additional costs for training and other organization 14

change management activities. 15

3. Scope Cost Management Tool (SCMT) 16

Table V-25 Scope Cost Management Tool118



The Scope and Cost Management Tool (SCMT) is foundational to the Work 18

Management process as it will provide a standard method of estimating the scope and costs of a project 19

throughout its lifecycle. An improved scope estimating solution is required for SCE to accurately 20

estimate the cost for all T&D and FERC 1000 projects, while managing to forecasts (via Portfolio 21


CIT-00-DM-DM-000093 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 2.00 3.00 - - 5.00


Recorded Forecast

81

Management) in a consistent manner. SCMT will allow for estimation accuracy commensurate with the 1

phase of the project and level of engineering estimate available. As large capital projects progress from 2

initial template-based estimates, to post job walks, or to detailed design,119 SCMT will provide the 3

ability to conduct scope and cost estimates with more detail and improved accuracy. 4

SCMT is intended for larger, non-programmatic projects120 that are included with 5

the Portfolio Management project, described above. SCMT will allow SCE to perform accurate scope 6

and cost estimating earlier in the project planning process. As an input into the Portfolio Management 7

solution, it will enable SCE to produce a comprehensive project forecast from inception to completion. 8

SCE’s existing process develops scope based on conceptual engineering prior to work order creation.121 9

However, revisions to scope and costs can also occur later in the planning and development phases of a 10

project. These revisions can cause impacts to the project schedule and budget when not properly re-11

estimated. 12

The SCMT project implements a solution that will: 13

1. Provide an accurate initial estimate of scope and cost using standard project 14

templates, then modified and revised as project scope is refined; 15

2. Provide a standardized platform for capital project scope estimation so they 16

are consistent and repeatable; 17

3. Expand the range of projects that can be estimated (e.g., major capital project, 18

programmatic project, interconnection, FERC 1000 projects, etc.); and 19

4. Assist in improved cost controls by providing a standard cost estimating 20

methodology. 21

SCMT will replace the disparate set of tools in use today with a single user-22

friendly platform. SCE uses a combination of software tools developed in-house, along with 23

commercially available products. As these products age, their performance diminishes. Their inability to 24

119 Detailed design estimates for work orders managed in SCE’s Design Manager would be produced from

within the Design Manager application. 120 Non-Programmatic work refers to projects whose scope is unique in nature and not repeatable. The

construction of a new large transmission line would be an example. 121 SCE’s Design Manager performs CU bases cost estimation for sub-transmission and distribution work, after

the work order has been pulled and engineered.

82

handle new types of work make them less useful. Replacing them with up-to-date technology will 1

alleviate the issues described above and provide a platform capable of meeting future needs. 2


Implementation of an industry-standard Scope and Cost Management Tool will 4

allow SCE to cost out the scope included in our current capital operating roadmap plans and more 5

accurately determine future spending needs. This allows SCE work managers and planners to perform 6

more accurate estimates for increasingly complex projects. This is accomplished by providing 7

predictable and consistent scope and cost estimating tools integrated to other enterprise work 8

management tools. Additionally SCMT will help enable SCE management to better analyze variances 9

between planned and actual project costs and scope, across all large capital projects. This analysis will 10

provide opportunities to improve the cost basis for future projects and improve our future project scope 11

costing and project controls capabilities. The estimates that the SCMT solution will provide are expected 12

to be an input into the Portfolio Management solution described above. 13

This project modernizes SCE’s tools and processes in use today and improves our 14

current capabilities in several major areas: 15

1. Ease of use – Effective project scoping and costing is an involved and often 16

complex process. The solution will allow for a simpler user experience by embedding complex rules and 17

processes into the workflow. It will also provide project scope and cost templates easily adjusted for a 18

project. 19

2. Consistency – A standardized tool will improve consistency of estimates and 20

therefore will cause improved accuracy. It will also provide more intelligence for users to understand the 21

variances between estimated and actual costs. 22

3. Performance – Besides not meeting SCE’s functional needs, the tools in use 23

are slow and cumbersome. The current tool can take up to 10 minutes or longer to calculate and display 24

results for large projects. Due to the architecture of the current tool, a user’s computer and its processor 25

are wholly occupied when running the SCMT analysis. The employee is left with little else to do other 26

than to wait for the transaction to complete. By using contemporary tools and technology with modern 27

architecture, the new solution will be more responsive and scalable. 28

4. Better Integration – As SCE moves toward a more robust EPC (engineering, 29

procure, construct) model, the project scope and costing tools must integrate with both internal and 30

external systems. This is especially true for major capital construction projects. 31

83

5. Flexibility – The current tools are limited in functionality to Transmission and 1

Distribution projects only. Since SCE is expanding the need for project scoping and costing accuracy 2

into other areas (e.g., Transmission Telecom), a more extensible and scalable tool will be required. As 3

other project types are incorporated into SCE’s IPSEC model, we can leverage the new Scope and Cost 4

Management Tool for them as well. By addressing the areas mentioned above, SCE will improve overall 5

capital project efficiencies, while providing more accurate capital spend plans. 6


In this rate case period, the total project costs are $5.0 million.122 The capital 8

forecast for this project was developed using SCE’s internal cost estimation model. This model utilizes 9

industry best practices and SCE subject matter expertise to estimate project cost components. SCE’s 10

forecast for this project includes costs for SCE employees, supplemental workers, and consultants, 11

software and vendor costs, and hardware costs. See this project’s workpaper for the cost breakdown 12

information. 13


Alternative 1: SCE considered enhancing the existing solution by 15

addressing the performance issues and extending the capabilities to include all project types. We did not 16

pursue this option because it would not be practical to meet our needs for better performance and would 17

require extensive work to integrate with other future systems. As stated above, the performance issues 18

are inherent to the system architecture and data model. Any enhancements would require significant 19

changes in the code and database, which would not be practical. Additionally, since the data model is 20

very different from other systems used in the IPSEC model, integration would be especially difficult and 21

prone to errors as there is no effective way to directly map across the data models. 22

Alternative 2: SCE considered replacing the existing solution by building 23

another customized solution to meet the needs of the SCMT scoping and costing functionality. We did 24

not pursue this option because it would not meet our requirements to use an industry-standard platform 25

and to minimize future maintenance costs. As stated previously, the existing tool was a custom-built 26


84

solution that was internally developed by leveraging the existing database from the MDI123 tool, and it 1

did not perform adequately. 2

Alternative 3: SCE considered leveraging its existing portfolio and 3

performing an integrated build by distributing the current functionality across other IT work 4

management initiatives, such as P6, SAP Project Systems, and Design Manager upgrades, if possible. 5

This would include a detailed gap analysis of the capabilities of these tools to the business requirements 6

and then develop processes for leveraging them. As reporting and notifications are major components of 7

SCMT, other systems must also be leveraged (e.g., Business Intelligence (BI) tools, eDMRM,124 8

SharePoint, Outlook). Similar to Alternative 1 above, modifying other systems to meet the requirements 9

was not pursued as there are significant gaps in functional requirements as well as dependencies on 10

systems and work processes that would require major changes (e.g., earlier creation of SAP work orders, 11

even for conceptual/unapproved project analysis). 12

4. Work Management Dashboard 13

Table V-26 Work Management Dashboard125



D.15-11-021 adopted the Work Management Dashboard project as part of SCE’s 15

2015 GRC. This project was delayed initially while the team assessed the project scope, business 16

impacts, and method of deployment. During the delay, it was discovered that additional business lines 17

123 MDI (Multiple Document Interface) is a Microsoft Windows programming interface for creating an

application that enables users to work with multiple documents at the same time. 124 eDMRM is Enterprise Document Management Records Management, a central records repository where

certain archived documents are stored. 125 Refer to WP SCE-04, Vol. 2 Bk B pp. 119-127.

CIT-00-SD-PM-000155 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 0.25 1.00 0.50 - - 1.75 Previous GRC Request* - - - 1.90 1.70 - - 3.60 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

85

within T&D were also using Access and Excel as business tracker databases. Using these tools require 1

extensive manual efforts, limit department-wide consistency, and introduce a large potential for errors. 2

The team considered various implementation approaches to minimize the organizational disruption that 3

would result from data conversion, interim parallel (redundant) processes, and a phased versus “big 4

bang”126 deployment. The project was subsequently delayed to accommodate the Work Management 5

Assessment (WMA) described in earlier sections. The WMA recommendation resulted in the scope for 6

the T&D Work Management (WM) Dashboard project being modified. Project Management capabilities 7

associated with project schedules for capital programs are now expected to be delivered through the 8

Portfolio Management solution. The Portfolio Management solution will capture major capital project 9

task information managed through several of the Access tracker databases and transition that data 10

collection as part of its project scope. The T&D WM Dashboard project will transition all other key 11

performance data into the proper enterprise tool to decommission the Access and Excel tracker 12

databases. The T&D WM Dashboard remaining project scope will focus primarily on reporting and 13

analytics. 14

The T&D WM Dashboard project will create an integrated work management 15

reporting and analytics solution to enable consistent planning, forecasting, and reporting across all T&D 16

capital and O&M work. This project will provide comprehensive reporting by integrating information 17

from the large capital work orders, capital maintenance work orders, customer-driven capital orders, and 18

all O&M orders. 19

To accomplish this, the WM Dashboard project will establish the reporting 20

database structure from which all integrated T&D management reports will be generated. The project 21

will select and implement existing enterprise technology, from which the standard reports will be 22

generated. This project will also provide training required by business analysts in order perform ad-hoc 23

reporting. 24


T&D does not have a reporting and analytics solution that provides the 26

organization with a comprehensive view of performance across all resources and types of work. The 27

organization relies on 18 Access databases and numerous Excel workbooks to fill this gap. These 28

126 “Big bang” deployments are deployments that typically include all users and all capabilities at one time.

86

databases are used by planners, project managers, and Resource Planning and Performance Management 1

(RPPM) to track the status of work order tasks (e.g., planner-related tasks such as design and 2

permitting), key work-order dates (e.g., customer commitment, regulatory compliance dates, and 3

material delivery dates), and contractor crew scheduling status, among other tasks. The current tools and 4

processes have the following issues: 5

• There is not a comprehensive view of T&D’s construction and maintenance 6

work, requiring multiple tools to gather and reconcile resources and work status. 7

• Resource forecasting and demand management across all T&D capital and 8

O&M work is therefore manually intensive. 9

• The current independent trackers have no automated work management 10

capabilities, are not integrated with core systems such as Design Manager, SAP or Click Schedule, and 11

require employees to manage work in multiple systems. This results in extensive duplicative data entry, 12

as the same data fields are entered into multiple trackers. This manual process of redundant data entry 13

can cause data quality and accuracy issues. 14

• The siloed tools are not developed according to standards and are difficult to 15

maintain. They are prone to lock up and crash, which results in lost time while the system and data are 16

restored. 17

(1) Benefits 18

The WM Dashboard provides an integrated system that will update 19

automatically to address the current data quality and accuracy issues, while eliminating duplicate data 20

entry. The WM Dashboard will also provide a comprehensive view of T&D’s construction and 21

maintenance work by replacing multiple tools with one integrated tool. The Dashboard tool will improve 22

management of resource demand and capacity planning for all capital and O&M work. The solution will 23

provide near real-time access127 to data; allow reporting based on current data; and provide enterprise 24

standard technology to support operation activities. 25


The first component of the WM Dashboard project is the development of a work 27

management dashboard that provides an integrated view of work across departments at a level of detail 28

127 Near real-time in this case pertains to the ability to report on new contractor-provided data multiple times

daily as needed.

87

that is not available today. Through this functionality, SCE can report on department employee 1

utilization, department employee capacity, work progress, work volume by status (completed, in-flight, 2

etc.), work volume by department, and by work type. 3

The second component will decommission the Access databases by replacing 4

tracker task data with equivalent data from core enterprise applications. This will include further 5

leveraging applications within the existing SCE software portfolio. The second component will evaluate 6

and select the best appropriate technologies to address any remaining capability or performance 7

requirements. These will be integrated with the enterprise work and portfolio management solutions. 8

This phase will also deliver improved forecasting and demand management 9

capabilities and resource capacity planning. 10

This project is scheduled to begin in October 2016 and be completed in June 11

2018. In this rate case period the total project costs are $1.8 million.128 The capital forecast for this 12



project includes costs for SCE employees, supplemental workers, and consultants, software and vendor 15

costs, and hardware costs. See this project’s workpaper for the cost breakdown information. 16


Alternative 1: SCE considered not pursuing this project and instead 18

continuing to use the current Access and Excel tools to manage workflow. However, if we chose not to 19

pursue this project, then this would address none of the existing issues or achieve the benefits described 20

above. Continued use of the Access database would result in continued disruptions to the business, as 21

the systems are unstable and experience frequent outages due to the volume of users trying to work in 22

them. Additionally, due to the instability of the databases, SCE is not building new data fields into the 23

systems. This is resulting in the creation of even more disparate methods of tracking additional data and 24

perpetuating the operational inefficiencies of managing information and inability to share common data 25

across organizations. 26

Alternative 2: SCE considered procuring a COTS solution to meet the 27

transmission and distribution work management needs. We did not pursue this option for the reason that 28


88

it would not meet our requirements, nor leverage the current technology footprint. Since there is no 1

commercially available product that currently meets all of SCEs business requirements, any selected tool 2

would still require effort to configure and develop the code necessary to integrate with SCE’s existing 3

enterprise work management tools. In addition, adding a new tool into SCE’s portfolio would add new 4

complexity and additional cost to maintain. 5

5. Transmission Telecommunications Work Order Lifecycle 6

Table V-27 Transmission Telecommunication Work Order Lifecycle129



The Transmission Telecommunication (Transmission Telecom) group within 8

T&D is responsible for the design, construction, and operations of an extensive telecommunications and 9

fiber-optic network that supports critical electric utility operations and business systems. The processes 10

used by the Transmission Telecom group are similar to those used throughout T&D. However, the 11

Transmission Telecom group does not use the same project and work-management tools used by the rest 12

of T&D for end-to-end planning, design, scheduling, field execution, and work order closure. This 13

Transmission Telecom Work Order Lifecycle project will extend existing work management tools used 14

by the rest of T&D (e.g., Design Manager, Graphical Design Tool, Click Schedule, and Powerplan) to 15

the Transmission Telecom group. 16

The Transmission Telecom group will leverage the existing Design Manager 17

(DM) core platform to: (1) Manage the assignment of work to the planning staff and measure 18

corresponding employee performance, (2) Apply the proper pricing methods to determine work order 19

estimates, (3) Verify that work order designs are completed and that material is appropriately selected 20

129 Refer to WP SCE-04, Vol. 2 Bk C pp. 128-136.

CIT-00-DM-DM-000095 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - - 2.00 2.00 - 4.00 Previous GRC Request - - - - - - - -

Recorded Forecast

89

according to job type, and (4) Confirm that the proper documentation, permits, and notices are achieved 1

and secured prior to construction. 2


Current Transmission Telecom processes consume significant effort and time 4

because data and information for work management processes are dispersed across several computer 5

systems. These systems are maintained by the business unit and therefore the required manual 6

configurations and regular maintenance must be performed by a few individual employees within the 7

Transmission Telecom team. This results in staffing inefficiencies as employees balance core business 8

functions with additional system support duties. It also presents risks to data quality and consistency 9

when the organization experiences employee turnover. Because the systems used by the Transmission 10

Telecom group are not consistent with the rest of the T&D organization, the data and information the 11

systems produce is also inconsistent. This can create duplication, operational misalignment, and 12

additional cost to how we approach work management across the T&D organization. 13

(1) Benefits 14

The Transmission Telecom Work Order Lifecycle project is key to the 15

organization’s ability to operate efficiently and productively. The project will align work order processes 16

and information with the rest of T&D by having work created and managed consistently across the 17

organization. Use of these standard technology platforms will better enable SCE to manage cost, 18

schedule, and risks through accurate initiation, planning, scheduling, and closure of work performed by 19

Transmission Telecom. Moving Transmission Telecom work onto a common T&D technology platform 20

also enables the Portfolio Management and Work Management Dashboard solutions to be extended to 21

include Transmission Telecom without customization. 22

This effort will reduce reliance on manual processes and provide an 23

integrated system for Transmission Telecomm processes. The effort will also leverage current enterprise 24

technology to capture work order transaction information (e.g., transaction time stamps, 25

created/modified employee data) that must otherwise be manually tracked and managed. 26

i. Work Order Planning 27

This project will extend the Design Manager (DM) application, 28

discussed further in this testimony, to the Transmission Telecom group. The Transmission Telecom 29

group will leverage the existing DM core platform to: (1) Manage the assignment of work to the 30

planning staff and measure corresponding employee performance, (2) Apply the proper pricing methods 31

90

to determine work order estimates, (3) Verify that work order designs are completed and that material is 1

appropriately selected according to job type, and (4) Confirm that the proper documentation, permits, 2

and notices are achieved and secured prior to construction. Use of DM by this group will provide 3

visibility of in-progress and completed work. The assignment and tracking of work orders and requests 4

through DM will provide a platform for communication and collaboration between Transmission 5

Telecom and other T&D departments. 6

ii. Graphical Design 7

SCE’s current Graphical Design Tool (GDT) is used by T&D 8

distribution and sub-transmission planners and provides a graphical design platform upon which these 9

individuals draft work order construction diagrams and maps. GDT also provides a planner or designer 10

the ability to capture and relay information related to safety, standards, and construction methods to the 11

field crew executing the work. Use of GDT by Transmission Telecom will: (1) Provide a standard 12

platform for the drafting and design of work orders, (2) Verify that drafts and designs comply with 13

standards and policy, (3) Confirm that appropriate material is selected for efficient construction, and (4) 14

Enable integration of work order designs with the Comprehensive Geographical Information System. 15

iii. Scheduling 16

SCE’s use of an automated scheduling tool for T&D work order 17

field execution aligns resources to planned units of work. The resulting work assignments are then 18

dispatched to crews. This capability allows the T&D organization to optimize work schedules for field 19

crews. Transmission Telecom has no similar scheduling system and relies on manual management of 20

resources to accomplish work. This project will extend SCE’s scheduling COTS tool, discussed later in 21

this testimony, to the Transmission Telecom group. Providing Transmission Telecom a standard COTS 22

platform for the scheduling and dispatch of work will make the scheduling process more efficient. The 23

ability to match available resources to planned work will cause the establishment of full and stable work 24

order schedules, which will bring productivity gains in the field execution of the work order. 25

iv. Execution & Closure 26

Leveraging much of the existing configuration for T&D work 27

orders will allow Transmission Telecomm to follow the standardized work order closure process and 28

take advantage of the existing integration between SAP and Powerplan. This approach will also result in 29

91

improvements to the closure process, as the work order accountants will no longer have to accommodate 1

exception processes for telecomm work orders. 2

SCE’s Field Accounting Organization (FAO) continues to pursue 3

opportunities to close work orders faster and more efficiently, while maintaining quality and completing 4

other core duties. By using the same technology as the other organizations, SCE can reduce process 5

inefficiencies resulting from different technology and manual processes.130 6


This project will extend existing technology platforms used by T&D to support 8

the Transmission Telecom group manage their end-to-end work processes. Transmission Telecom will 9

leverage DM and GDT for the Initiation and Planning phases. The tools will improve the work order 10

management process, work order creation, work order design drafting, and the pricing of work to be 11

executed. Click Schedule will support the scheduling aspect of the work order process. This will support 12

field crews to be appropriately equipped and available for the execution of the scheduled work. SAP and 13

Powerplan will verify proper completion and closure of the work order, and maintenance and archival of 14

related documentation. This project will implement system configuration changes to the existing 15

architecture already in place and develop software to support regulatory and compliance-related 16

activities. 17

The project is scheduled to start in the second quarter of 2018 and be completed 18

by the fourth quarter of 2019. The capital forecast for this project was developed using SCE’s internal 19

cost estimation model.131 This model utilizes industry best practices and SCE subject matter expertise to 20

estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 21

supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 22

workpaper for the cost breakdown information. 23


Alternative 1: SCE considered not pursuing this project. However, if we 25

chose not to pursue this project, it would result in significant levels of redundant work across multiple 26

departments, higher operating costs for projects requiring collaboration, inability to streamline work 27

130 Field Accounting Organization (FAO) can now receive inputs from one consistent process, rather than a

separate process unique to Transmission Telecomm. 131 Refer to WP SCE-04, Vol. 2 Bk B pp. 136.

92

orders, inability to maintain and archive designs and records, and the inability to support audit-related 1

activities in a timely manner. 2


needs of the Transmission Telecom group. We did not pursue this option because it would not meet our 4

needs unless it was customized to specifically address the Transmission Telecom requirements. In 5

addition, this would require introducing another application into our environment, require new 6

interfaces, and also add further costs for licenses and on-going maintenance. Due to the high complexity, 7

cost, and risk, SCE elected not to pursue this option. 8

6. Click Schedule Refresh Release 1 & 2 9

Table V-28 Click Schedule Refresh Release 1&2132



D.15-11-021 adopted the Scheduling Refresh Project to refresh the outdated 11

scheduling application and extend the scheduling capabilities to other T&D business lines. 12

Subsequently, the project scope was divided into two releases to minimize the daily operational impact 13

of new functions being introduced to the Transmission and Distribution organization. 14

The Scheduling application was initially deployed in 2007 to T&D Distribution 15

groups including Resource Managers, Schedulers, and their support staff as part of the Enterprise 16

Resource Planning (ERP) program.133 In 2010, also as part of the ERP program, the user base was 17

expanded to include the Transmission, Substation, Grid Operations, and Power Delivery Resource 18

Planning and Performance Manager (RPPM) organizations. Additional planned users include 19

132 Refer to WP SCE-04, Vol. 2 Bk B pp. 137-140. 133 SCE’s Enterprise Resource Planning (ERP) program was implemented in 2010.

Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - 0.56 1.52 2.50 3.50 0.50 - 8.58 2015 GRC Authorized* - - - 5.40 2.50 - - 7.90 2015 GRC - Original Request - - 0.38 5.40 2.50 - - 8.28 *The Commission adopted SCE's request for this project in D.15-11-021. Multiple WBS elements include: CIT-00-DM-DM-000030 and CIT-00-SD-PM-000232.

Recorded Forecast

93

construction coordinators, damage assessment teams, Troublemen, dispatch coordinators, and mutual 1

assistance crews. 2

SCE started the project in 2014 using O&M funds to perform a preliminary 3

impact assessment. SCE engaged an experienced scheduling consultant to recommend an 4

implementation approach. SCE then undertook a competitive selection process to select the most 5

qualified implementation vendor. The assessment took longer than anticipated and delayed the start of 6

the project until 2015. 7

Release 1 refreshes hardware, updates the scheduling software version and 8

database, and updates interfaces to SCE systems. This release provided “like for like”134 functionality to 9

the users. Release 2 delivers additional capabilities such as automated dispatch, crew boards, integration 10

with crew call-out system, and integration with SCE’s Comprehensive Geographical Information 11

System (cGIS). 12


T&D’s scheduling application is four versions behind the current vendor product 14

and is no longer supported due to its obsolete technology foundation. Our contract with the vendor 15

requires us to stay within two major versions of the vendor’s most recent commercial release. If we fail 16

to stay within two major versions, our contract requires that we purchase new licenses if we refresh the 17

application. SCE did not complete the required major version upgrades of the software due to the lack of 18

additional features that would provide additional operational improvements for T&D and because the 19

newer versions would require substantial effort to install. But the vendor is waiving this license 20

repurchase provision if we refresh now. Refreshing the scheduling application now will allow us to 21

continue having vendor support. 22

The Scheduling Project Release 2 is scheduled to start in 2017 and will complete 23

in 2018. The project will address additional gaps we have found with the current version and provide 24

additional capabilities. 25

134 “Like for Like” is a term used within SCE to describe when an older or unsupported application is replaced by

a newer, more technologically advanced application, architecture, and interface, that provides the same functionality, use, and results.

94

(1) Benefits 1

By completing Scheduling Project Release 2, SCE will be positioned to 2

gain additional benefits by: (1) Reducing multiple short-term scheduling tools across T&D, (2) 3

Providing a scheduling solution for work groups in other SCE Operating Units, (3) Improving 4

integration with work management, outage, storm, mapping, and ERP solutions, and (4) Eliminating 5

custom code “add-ons.” SCE will also avoid a larger and more costly project by leveraging the 6

Release 1 investment (see Alternative 2 below). 7


The first release was started mid-2015 and is targeted to complete in 2017. The 9

second release is scheduled to start in 2017 and complete in mid-2019. In this rate case period the total 10

project costs are $8.6 million.135 The capital forecast for this project was developed using SCE’s internal 11

cost estimation model. This model utilizes industry best practices and SCE subject matter expertise to 12





Alternative 1: SCE considered keeping the existing solution without 17

modification. We did not pursue this option because we would have to continue with the current labor-18

intensive and inflexible process, work around technology obsolescence, and continue with limited 19

visibility to field work orders status and crew availability and with high maintenance and support 20

overheads due to process gaps. This option also leaves SCE without necessary vendor support. 21


procuring another vendor’s COTS product. The original scheduling project was an approximately $34 23

million effort and spanned five years. A new COTS system would require an investment in time and 24

money to provide new license purchases, integration, and build interfaces for the new application with 25

our other existing software applications. 26

Neither of these options address the current issues described above in the 27

most cost-effective way. 28


95

7. Vegetation Management 1

Table V-29 Vegetation Management136



D.15-11-021 adopted the Vegetation Management (VM) project as part of SCE’s 3

2015 GRC. SCE subsequently changed our contractor sourcing strategy and developed new vendor 4

management requirements. To address the changes in strategic direction, SCE conducted research to 5

determine the best approach and software tools to enable the new VM capabilities. To minimize the risk 6

of developing a solution that did not support the new processes, the project did not begin development 7

until the new processes were fully defined in 2015. 8

The VM project will consist of three phases: (1) deployment of a mobile software 9

and hardware solution to enable data collection in the field, (2) implementation of COTS VM software, 10

and (3) integration to SCE systems such as SAP and cGIS. The project is scheduled to be completed in 11

2017. 12

The VM project implemented a new mobile field collection device in late 2015. 13

This phase of the project was funded with O&M. This device is in use by pre-inspectors, tree-trimmers, 14

and SCE VM employees. In the upcoming phases, the VM project will also implement new systems and 15

analytic tools used to plan and analyze VM work processes such as tree trimming, tree removal, and 16

weed abatement. The VM project will replace the current manual process, which is paper-intensive and 17

outdated. 18

Since the last GRC, SCE has revised its implementation approach, which has 19

resulted in a reduced forecast compared to SCE’s 2015 request.137 SCE recognized that some relatively 20

straight-forward changes in current processes would allow for a COTS solution, without compromising 21

136 Refer to WP SCE-04, Vol. 2 Bk B pp. 141-148. 137 Refer to WP SCE-04, Vol. 2 Bk B p. 147.

CIT-00-SD-PM-000154 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total

Recorded / Forecast - - - - - 2.00 5.70 - - - 7.70

Previous GRC Request* - - - 0.90 4.00 4.80 - 9.70 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

96

the safety and reliability objectives. This also allowed SCE to avoid the costs of customizing a COTS 1

product. In addition, SCE elected to use a cloud-hosted solution, instead of building up the additional 2

infrastructure within SCE’s environment. Both these decisions resulted in reduced project costs. 3


The current VM process involves manually entering information from paper 5

forms into an Access database. The current Access front end and the SQL database used for VM are 6

antiquated and are operating near peak capacity. The VM group manages regulatory compliance of 1.5 7

million trees and 600,000 to 700,000 annual tree trim records. Managing this large volume of work 8

within the current process and tools has inherent risks. These risks include keeping data current while 9

processing a large backlog of paperwork, and controlling data errors resulting from manual data entry. 10

By replacing the current paper-intensive process with a digitized, map-based 11

system, and providing the data and tools to perform better analytics and work management, the VM 12

project will: (1) Increase SCE’s ability to track and report on the quality of its contractors’ performance, 13

(2) Increase the ability to prioritize work, including the ability to prescribe work proactively instead of 14

reactively, which will help expedite the removal of potentially dangerous trees, and (3) Reduce the risk 15

of non-compliance with requirements from federal, state, local and environmental agencies (including 16

General Order (GO) 95, Rules 35 and 37, and Public Resources Code (PRC) 4292 and 4293). This will 17

be achieved by integrating the VM with our Comprehensive Geographic Information System (cGIS) to 18

enable monitoring of trees in areas with drought conditions, and align prioritization of tree trimming 19

regarding the distribution and transmission facilities. 20


The project will be implemented in three phases. Phase One (deployed in 2015) 22

focused on the deployment of the field data collection software and hardware.138 Phase Two will 23

configure and deploy a COTS VM system, and Phase Three will complete the project integration to SCE 24

systems such as SAP and cGIS. Total project capital costs are $7.7 million.139 25


estimation model. This model utilizes industry best practices and SCE subject matter expertise to 27

138 The preliminary planning activities of this VM project were funded through O&M costs, and therefore are

requested for recovery in this GRC. 139 Refer to WP SCE-04, Vol. 2 Bk B p. 148.

97






modification. We did not pursue this option because it would not address the current issues or achieve 6

the benefits described above. 7

Alternative 2: SCE considered building a customized solution to meet the 8

needs of the VM group. We did not pursue this option because this would require significant investment 9

of time and effort to build and test a new application from scratch, which would not be cost effective for 10

SCE to develop. This alternative would subsequently add additional costs and complexity to the 11

maintenance of the application. 12

8. Pole Loading Application Replacement Tool 13

Table V-30 Pole Loading Application Replacement Tool140



D.15-11-021 authorized the Pole Loading Application Replacement (PoLAR) 15

project. General Order 95 (GO 95) sets forth rules for the design, construction, and maintenance of 16

overhead lines. GO 95 specifies that, at the time of installation, a newly constructed utility pole must be 17

built with an acceptable safety factor. Calculations are used to determine what the safety factor is—this 18

calculation is called pole loading (i.e., determining the load and its effect on the pole). In 2012, SCE 19

began the PoLAR project to deliver a long-term solution that enables T&D personnel to perform pole 20


CIT-00-SD-PM-000137 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - 1.09 3.39 4.28 3.64 2.44 - - - - 14.83 2015 GRC Authorized* - 1.09 3.39 2.74 - - - 7.22 2015 GRC - Original Request - 1.09 4.33 2.74 - - - 8.16 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 costs.

Recorded Forecast

98

loading analysis that meets safety and regulatory requirements, while performing effectively within a 1

dynamic technical landscape. Besides meeting core requirements to enable regulatory compliance, the 2

PoLAR project will also improve the end-to-end work order process through integration of several 3

workflow management applications (Design Manager/SAP) to optimize data transfer, pole load control 4

processes, and reporting capabilities. 5

In December 2013, the PoLAR scope was increased to deliver additional 6

capabilities to enable SCE’s new Pole Loading Program (PLP). Through PLP, SCE is performing a 7

systematic assessment of all existing poles in SCE territory, as described in SCE-02 Volume 9.141 8

Because of this new organization and process, PoLAR expanded from replacing the pole-loading tool, to 9

a larger effort to enable the PLP. This project has implemented several new processes and technologies 10

to enable this significant increase in the volume of all pole-related transactions (e.g., inspection through 11

construction and closure). 12

While a significant portion of the PoLAR project has been implemented and is 13

operational, certain business critical capabilities remain to be delivered. Emergent business priorities, 14

such as the initiation of the PLP, resulted in shifts of the original implementation schedule, resulting in 15

the delay in delivering the capabilities. These requirements were originally written in 2012, and the 16

development work that needs to occur to implement them has increase, because the business processes 17

have grown in complexity over the past 4 years. SCE has incorporated these additional technical 18

requirements into the project’s scope and determined that additional costs would be required to support 19

these new requirements. Additional capabilities included in the revised scope include new automation 20

for certain steps in our PLP quality control processes, and integration with cGIS to automate pole-related 21

data submissions to the PLP assessment vendor. SCE is also supplementing the original scope of work 22

to add more integration, data touch points, and other system-related requirements to accommodate these 23

additional capabilities. SCE will complete the final phase of the PoLAR project in 2016. 24

141 The Commission adopted SCE’s Pole Loading Program in D.15-11-021, p.123.

99

b) Recorded Costs and Forecast 1

SCE has recorded $12.4 million through the end of 2015 and expects to spend an 2

additional $2.44 million in 2016.142 This increase above authorized levels is a direct result of the 3

increase in scope described above. 4

The remaining capital forecast for this project was developed using SCE’s 5

internal cost estimation model. This model utilizes industry best practices and SCE subject matter 6

expertise to estimate project cost components. SCE’s forecast for this project includes costs for SCE 7

employees, supplemental workers, and consultants, software and vendor costs, and hardware costs. See 8

this project’s workpaper for the cost breakdown information. 9

9. Design Manager (DM) Refresh 10

Table V-31 Design Manager Refresh143



D.15-11-021 approved the Design Manager (DM) Refresh project. The current 12

forecast of $3.7 million is an increase of $1.0 million relative to our 2015 GRC. This project began in 13

August 2013 and will be completed in 2016. The capital forecast for this project includes $1.0 million in 14

project team costs for SCE employees, supplemental workers, and consultants, and $2.7 million in 15

software and vendor costs. No hardware costs are included for this project. 16

SCE’s DM application is used by planners for distribution, transmission, and 17

substation work. DM acts as the intermediary between the Graphical Design Tool (GDT) application 18

and SAP, and it is the pricing and estimating tool used by SCE planners in both transmission and 19

distribution and by substation engineers. DM is an eight-year-old, developed-in-house system. DM is 20

142 Refer to WP SCE-04, Vol. 2 Bk B p. 153. 143 Refer to WP SCE-04, Vol. 2 Bk B pp. 154-158.

CIT-00-SD-PM-000152 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - 0.90 1.15 1.62 - - - - 3.68 2015 GRC Authorized* - - - 2.20 - 2.20 Previous GRC Request* - - 0.50 2.20 - - - 2.70 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 costs.

Recorded Forecast

100

used by planners in work creation, reporting, compatible unit selection, and cost estimating. As the 1

usage of this tool has expanded across multiple business lines, several gaps have been identified within 2

the existing functionality. The current DM refresh project will update the software and hardware 3

infrastructure, and it will add functionality to enable better overall reporting and usability. 4

The DM project will change the current application from a front-end application 5

loaded on employee computers to a web-enabled application. This tool provides the ability to work from 6

mobile devices, such as tablets, reduces contractor connectivity issues by freeing them to upgrade their 7

hardware, while using more advanced processing. In addition, this project will improve the integration 8

between DM and Graphic Design Tool (GDT) project discussed in the following section. 9

Besides updating the DM application, the DM refresh project will replace the 10

current reporting functionality of DM with SAP Business Intelligence (BI) reporting to provide 11

improved reporting. This will give SCE the ability to provide employee performance reports, ad-hoc 12

reports, weekly status reports, and work order aging reports. In summary, the DM Refresh will address 13

the issues with the current system that include: (1) lack of mobility while using the DM application 14

described above, (2) system performance issues described above, (3) limited reporting capabilities 15

described above, (4) lack of ability to support other projects including WM Dashboard and Electronic 16

Work Order Package, (5) limited visibility of work management and tracking, (6) contractor 17

connectivity issues due to requirements to use older hardware and software platforms, (7) the 18

requirement for the user to create notifications for system issues by calling SCE’s help desk and 19

explaining the issue, and (8) limited ability to perform root-cause analysis for system defects. 20

DM Refresh will reduce risk by providing the technology updates needed to 21

mitigate information security exposures, and it will reduce the possibility of application failures 22

resulting from outdated or unsupported vendor technology. The DM Refresh project will provide 23

solutions for many of the existing architectural and usability issues that slow down or cause repetitious 24

tasks for SCE planners and designers. 25

b) Recorded Cost and Forecast 26

When the DM Refresh project began in 2013, SCE selected the vendor that 27

provided the best proposal for executing the project, considering cost and schedule. During the analysis 28

period of the project, however, several issues and gaps were discovered in the vendor’s understanding of 29

the application and the scope of the project. At this time, SCE was at a decision point to either continue 30

with the current vendor, or to close the analysis period and engage a different vendor for the project. 31

101

SCE elected to engage a different vendor for a couple reasons: (1) the new vendor we engaged was the 1

same supplier that constructed the original version of Design Manager and was already an expert in the 2

system, and (2) this vendor was also recently selected as one of SCE’s managed service providers 3

(MSPs). By using one of SCE’s MSPs for the project effort, it would minimize the effort of transitioning 4

the project (once completed) to the maintenance mode, since it is the same vendor providing both 5

services. While this change in vendor partner resulted in the identification of additional costs during the 6

analysis phase, it also mitigated the risk of finding more impactful gaps much later in the development 7

processes. 8

The remaining capital forecast for this project was developed using SCE’s 9

internal cost estimation model.144 This model utilizes industry best practices and SCE subject matter 10



this project’s workpaper for the cost breakdown information. 13

10. Graphic Design Tool (GDT) and Tract Deployment Refresh 14

Table V-32 Graphic Design Tool and Tract Deployment Refresh145



The Commission adopted the Graphical Design Tool (GDT) Refresh, and the 16

GDT for Tract Planners, Underground Drafting and Design Group (UGDDG) projects,146 as part of 17

SCE’s 2015 GRC. The GDT Refresh was originally planned to begin in 2015. During our planning 18

process it was determined that delaying the effort to incorporate a later vendor product release would 19

144 Refer to WP SCE-04, Vol. 2 Bk B p. 158. 145 Refer to WP SCE-04, Vol. 2 Bk B pp. 159-168. 146 See SCE-05, Vol. 2 from A.13-11-003.

CIT-00-DM-DM-000023 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 1.50 3.50 1.60 - 6.60 Previous GRC Request* - - - - 0.95 3.25 - 4.20 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

102

allow for SCE to combine the two projects thus reducing change impacts to our systems and business 1

processes. The revised approach modified the project plan to begin in late 2016. 2

The GDT application is a system used by SCE’s distribution and sub-transmission 3

planners. GDT was originally implemented with the Design Manager tabular application to provide 4

graphical design capability to planners. GDT was first implemented in 2009 for distribution planners, 5

and subsequently integrated in 2010 with SAP, which expanded the tool’s availability to transmission 6

estimators. 7

During the original GDT project in 2008-2010, SCE partnered with the software 8

vendor to develop several customized capabilities to enhance the COTS software application. The 9

vendor has since recognized the value of these capabilities and has incorporated them into its newest 10

commercial product. By adopting this new version, SCE will eliminate the additional effort required to 11

maintain the customized software and will benefit from the capabilities now supported by the vendor. 12

SCE has also determined that combining the GDT Refresh and GDT for Tract Planners and UGDDG 13

would be a least-cost option for adopting the newest vendor product and would expand its use to new 14

user groups. 15

Combining these two efforts into a single project will reduce the risk of managing 16

and implementing multiple system changes and complex integrations simultaneously under separate 17

initiatives. The single project planning approach will provide consistency of resources and project 18

management activities across both phases while optimizing efforts to communicate and train employees 19

on the new tools. 20


The GDT Refresh and deployment of the GDT to the UGDDG is essential for the 22

continued use of the existing software. The current version is no longer supported by the vendor. In 23

addition, deployment of the application to the Tract and Underground Drafting planners is necessary to 24

move all planners and designers onto a centralized common design platform. The anticipated new user 25

groups use an antiquated, non-vendor-supported system to complete their design work. These groups 26

support other design groups by producing graphical designs for their work. The inconsistency of 27

applications used between multiple groups and among various resources within the design organization 28

causes difficulties in data sharing and limits the capacity to develop consistent work practices and 29

metrics. A common platform promotes efficient data sharing with no duplicate data entry or re-work 30

when projects are transitioned across groups within the same organization. 31

103


SCE forecasts $6.6 million to complete this project. The capital forecast for this 2

project was developed using SCE’s internal cost estimation model.147 This model utilizes industry best 3



costs, and hardware costs. See this project’s workpaper for the cost breakdown information. 6


Alternative 1: SCE considered not pursuing this project, and instead 8

continuing on the current version and delaying the refresh. Further delay is not recommended based on 9

the age of the existing application. Incorporating and maintaining old architecture creates complexities 10

when integrating into modern platforms and can add costs and delays to efforts such as operating 11

systems and server upgrades. The current version will be difficult for SCE to maintain, as it is no longer 12

supported by the product vendor. Delaying further will not address the business needs explained above. 13

Additionally, further delay poses a risk to procuring new licenses for a product upgrade. Our contract 14

requires that we purchase new licenses if we fail to maintain product versions within agreed-to limits. 15

Re-licensing risks additional increases to our forecasted costs. 16

Alternative 2: SCE considered separating the GDT Refresh and Tract 17

deployment efforts. This is not recommended because it will increase the cost and effort of redundant 18

training and communication activities. This approach would also increase significant risk of failure, due 19

to applying system changes multiple times, within the same application. 20


104

11. Consolidated Mobile Solution (CMS) 1

Table V-33 Consolidated Mobile Solution148



D.15-11-021 adopted the Commission adopted the Consolidated Mobile Solution 3

(CMS) project. CMS encompasses three components: (1) a collection of field computing devices (field 4

tools), (2) a mobile software application, and (3) required infrastructure that allows SCE’s field 5

employees to manage work even while not connected to the SCE network. The field tools consist of 6

ruggedized portable laptop computers and automated vehicle locating devices, all of which incorporate a 7

global positioning system. The mobile software application, which is installed on the ruggedized laptop, 8

enables the tool to function as a remote device with real-time capabilities to access asset and map 9

information. It also provides the capability to create edits for map updates, which are then processed by 10

back-office personnel using SCE’s Enterprise Resource Planning (ERP) system, Outage Management 11

System (OMS), and the Comprehensive Geographical Information System (cGIS) database. The 12

infrastructure required for the project includes telecommunications networks, servers, databases, and 13

system interfaces that pass information between the mobile application and supporting office systems 14

(e.g., ERP, cGIS, and OMS). CMS enables field personnel, system operators, and office workers to 15

work more efficiently, resulting in productivity benefits, enhanced employee safety, improved outage 16

responsiveness, and improved ability to meet compliance obligations. 17


To improve safety and reliability in maintaining our electrical system, SCE field 19

personnel must have access to the same applications, data, and maps as our office staff. The existing 20


CIT-00-SD-PM-000041 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 3.74 10.73 15.65 10.82 6.33 5.54 5.45 0.37 - - - 58.64

2015 GRC Authorized 3.74 10.73 15.65 10.82 5.42 - - - 46.37 2015 GRC - Original Request 3.74 10.73 15.65 9.00 7.03 - - - 46.16 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 costs.

Recorded Forecast

105

eMobile field tool is technically obsolete and is difficult to maintain. eMobile was an in-house-1

developed solution that consolidated several field tools. In eMobile, information transfer is a batch-2

process, not real-time, which leads to inconsistencies between field conditions and back-office data. It is 3

programmed in an older version of the programming language and operates on Windows XP. Windows 4

XP is out of normal support by Microsoft; this lack of support requires SCE to pay an additional support 5

cost of $0.59 million per year. The additional support costs are adjusted upwards each year. 6


The CMS project has deployed to approximately 700 T&D field users. 8

Deployment to the remaining 700 users’ devices is underway and planned to be completed by Q1 2017. 9

The CMS project has completed the application configuration, implementation, and deployment to T&D 10

Grid Operations (Streetlight team in 2013 and Substation Operators in 2015), Distribution organization 11

(Electrical System Inspections and Quality Assurance teams in 2014), and Substation Construction & 12

Maintenance organization (from 2015 to 2016). 13

In 2016 and 2017, the CMS project requires $5.82 million to complete the SCE 14

configuration of the application, implementation, training development, deployment and stabilization of 15

the remaining user groups (Grid Ops Troubleman, Distribution Construction & Maintenance (DC&M) 16

users, and Apparatus and Transmission). This will fully decommission the legacy eMobile application 17

and eliminate the continuing Windows XP support requirements, significantly reducing the operational 18

risks of relying on outdated and unsupported hardware and software for this highly utilized application. 19

SCE used a phased approach to roll out the solution to field users. Each roll out 20

was allowed to stabilize, while the lessons learned were applied to subsequent releases. One of the 21

lessons learned was that more end-user involvement in determining the functional look and feel of the 22

application was needed. This approach also resulted in the reworking of requirements to clarify 23

functions and a subsequent redesign of the software. The changes were larger than anticipated, resulting 24

in deployment delays and higher development costs. The software vendor had difficulty developing 25

solutions to meet the new requirements, which also contributed to schedule delays and resulted in 26

additional costs. However, the resulting delay has allowed SCE to deploy a solution that is higher 27

quality and rapidly accepted by the field users. 28

D.15-11-021 authorized total CMS project costs of $46.4 million. The total 29

project costs are now forecast to be $58.6 million. This increase in expenditures is a result of the 30

increase in scope described above. The remaining capital forecast for this project was developed using 31

106

SCE’s internal cost estimation model. This model utilizes industry best practices and SCE subject matter 1



this project’s workpaper for the cost breakdown information.149 4


SCE considered stopping development of this in-flight project due to the 6

extended schedule and higher than forecasted costs. From a technical risk assessment, the option to stop 7

the project and consider an alternate solution is not viable. While solutions are available in the market, 8

none acceptably meet SCE’s requirements without a significant amount of customization. When the 9

project started, none of the existing market vendors had a viable solution. Over time, these vendors 10

developed a comprehensive workforce solution. However, none of these vendors provide an integrated 11

work management and mapping capability similar to what is provided by CMS. 12

12. Field Tools Upgrade 13

Table V-34 Field Tools Upgrade150



D.15-11-021 adopted the completion of the Consolidated Mobile Solution (CMS) 15

project, to deploy approximately 1,400 field tools to T&D,151 as part of SCE’s 2015 GRC. The Field 16

Tools Upgrade builds on the CMS platform and enables SCE’s field personnel, system operators, and 17

office workers to improve employee safety, outage responsiveness, and SCE’s ability to meet 18

compliance obligations. 19

149 Refer to WP SCE-04, Vol. 2 Bk B p. 174. 150 Refer to WP SCE-04, Vol. 2 Bk B pp. 175-180. 151 See Consolidated Mobile Solution (CMS) Testimony for foundational project description.

CIT-00-DM-DM-000047 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total

Recorded / Forecast - - - - - - - 1.00 6.00 3.00 10.00 Previous GRC Request - - - - - - - -

Recorded Forecast

107


SCE’s focus on maintaining and improving the grid has grown over the last three 2

years. This growth has increased the need for SCE’s field personnel and system operators to work more 3

effectively and efficiently, and it has highlighted opportunities to improve how work is performed. 4

Work performance can be improved by optimizing work and resource 5

assignments across the IPSEC (Initiate, Plan, Schedule, Execute, and Close) process. This initiative 6

supports delivery of recommendations from the “Work Management Assessment” as described in the 7

work management testimony presented previously in this volume, and it improves work execution by 8

SCE’s field workers. 9


The Field Tools Upgrade project builds on the platform delivered by the CMS 11

project. It will enhance the solution to: (1) enable access of information from mobile devices such as 12

smartphones, iPads, and tablet devices, (2) enable location-based services to reduce the data footprint in 13

the Field Tool device, (3) allow contractors and mutual assistance crews to use the Field Tool in case of 14

a storm, (4) expand use of the Field Tool to other SCE Operating Units instead of buying or building a 15

comparable system, and (5) improve data quality, data consistency, and interface between the Field Tool 16

and SCE supporting office systems such as SAP and Comprehensive Geographical Information System 17

(cGIS). 18

The total project forecast is $10.0 million.152 The capital forecast for this project 19

was developed using SCE’s internal cost estimation model. This model utilizes industry best practices 20

and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 21


and hardware costs. See this project’s workpaper for the cost breakdown information. 23


Alternative 1: SCE considered not pursuing this project. While the 25

current Field Tool (CMS) is operational, our experience suggests that as additional assets are added to 26

the existing system and the user base expands, the potential for the system performance to degrade 27

increases. In addition, the enhanced mobility options such as tablets and smartphones could not be 28


108

provided to SCE’s field personnel. Mobile devices can improve efficiency, usability and user safety for 1

certain tasks such as inspections. 2

Alternative 2: SCE considered replacing the existing field tool solution 3

by building or buying a customized solution. This option was rejected because it would not be cost-4

effective to create such a solution, it would cause a redundant application, and it would create additional 5

maintenance costs to the existing system. 6

13. Enhanced Business Resiliency for Energy Management System (EBR) 7

Table V-35 Enhanced Business Resiliency for Energy Management System153



SCE’s Energy Management System (EMS) supports the reliable operation of the 9

entire SCE power system including the distribution and transmission networks, and is itself supported by 10

primary and back-up systems at two separate SCE locations. This configuration provides for redundancy 11

and protection from physical concerns such as an earthquake or other conditions that would require a 12

back-up site to be activated and staffed. However, both EMS systems are connected to, and run off of, 13

the SCE IT network, which may pose a cybersecurity risk. The Enhanced Business Resiliency for EMS 14

Project establishes a third, off-network back-up site to protect against threats, primarily cyber-security 15

threats, to the existing two EMS systems. 16


A complete loss of EMS availability would affect the Grid Control Center and the 18

ability of SCE’s 14 switching centers to monitor, control, and optimize the power grid and detect and 19

respond to outages. Any outages while EMS is not available pose a significant risk to public safety, SCE 20

personnel, and external SCE customers, including the CAISO and Peak RC (Reliability Coordinator). 21


CIT-00-DM-DM-000051 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.00 4.00 - - 7.00 Previous GRC Request - - - - - - - -

Recorded Forecast

109

The system and the aforementioned configuration do not provide complete protection from new and 1

emerging security threats. These threats present a potential vulnerability. Besides having other security 2

controls, the additional site would be off of the network for a majority of the time, which reduces the 3

risk of being compromised if the primary and secondary on-network locations are attacked. The tertiary 4

EMS instance could start up within minutes of the primary and secondary locations becoming 5

unavailable. 6

(1) Benefits 7

EMS is used by over 200 system operators and engineers, both at the Grid 8

Control Center (GCC) and SCE’s Switching Center. These personnel rely on the EMS to provide 9

monitoring and controlling functionality and to provide analytical capabilities and reliability solutions. 10

The EMS Real Time Contingency Analysis screens outages and outage combinations and will alert 11

operations if any of the outages being prescreened will have a reliability impact on the system. This 12

critical analysis capability would not be possible if EMS were not available. In addition, it would not be 13

possible for the CAISO or the Peak RC154 to perform this analysis on the SCE grid. SCE outages 14

comprise approximately 50% of CAISO’s contingencies and 30% of Peak RC, which illustrates the 15

potential impact to the western grid. 16

SCE’s EMS system is critical to the reliability of not only the SCE system, 17

but the Western Interconnection. A third instance of EMS that is not attached to the network would 18

increase SCE’s ability to function in certain threat scenarios (e.g., cyber-attack, malware infection). SCE 19

customers and system reliability would benefit from the availability of a third back-up system. The 20

approach of supplementing an online configuration with an instance that stays disconnected from the 21

network for a majority of the time is consistent with cybersecurity best practices, in which high-value 22

targets like “Key Infrastructure” stay disconnected to keep the attack surface as small as possible. 23


This project will provide a third back-up, standalone system that will not be part 25

of the site synchronization scheme between the existing EMS Primary and Backup systems. The system 26

will receive configuration updates and manual entries when it connects to the network periodically. 27

Once activated, the operators could start the system up with current configuration and synchronize with 28

154 PEAK RC (Peak Reliability Coordinator) provides real time monitoring of the Western Interconnection.

110

manual updates within a reasonable time (expected to be within an hour). This is the preferred method, 1

as it poses the least risk of being compromised by cyber events, and with proper site selection could 2

provide the additional protection against major catastrophic events such as earthquakes. 3

The total project forecast is $7 million.155 The capital forecast for this project was 4






Alternative 1: SCE considered building a network-isolated mobile system 10

housed in a vehicle, potentially a van or an RV. The vehicle would have an independent source of 11

electrical power (generator) and the means to communicate. The system would be isolated and 12

periodically synchronized with the active EMS system. We did not pursue this option due to the inherent 13

space limitations of a mobile option. Due to the limited space, the system would have to be a scaled-14

down version of the primary/backup EMS with limited functionality. 15

14. Comprehensive Situational Awareness for Transmission (CSAT) Phase 1 16

Table V-36 Comprehensive Situational Awareness for Transmission156



D.15-11-021 adopted the Comprehensive Situational Awareness for Transmission 18

(CSAT) project in SCE’s 2015 GRC, where the project was referred as “Advanced Phasor Data 19

155 Refer to WP SCE-04, Vol. 2 Bk B pp. 186. 156 Refer to WP SCE-04, Vol. 2 Bk B pp. 187-193.

CIT-00-DM-DM-000085 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 2.00 4.00 8.00 8.00 22.00 Previous GRC Request* - - - 1.20 7.70 4.20 - 13.10 *The Commission adopted SCE's request for this project in D.15-11-021 under the Advanced Phasor Analytics project name.

Recorded Forecast

111

Analytics” under the Phasor program. The objective of the CSAT program is to provide the ability for 1

Grid Control Operators to assess the status of the entire transmission system at a glance and provide 2

quick access to detailed data and robust analytics to make more informed decisions during critical 3

operational periods. This project was scheduled to be deployed from 2014 through 2016. 4

SCE did not launch the Phasor Analytics project as proposed in our 2015 GRC 5

Application. The delay in the CSAT project launch was a result of the extended deployment and 6

stabilization157 of the Phasor project. CSAT requires Phasor data, and the effectiveness of the analytics 7

engine depends on the deployment of several strategically located time-synchronized Phasor 8

Measurement Units (PMUs) to provide improved wide area situational awareness data. SCE will have 9

deployed a sufficient number of PMUs by 2018 to enable the analytics engine to deliver reliable and 10

accurate real-time information. 11


SCE requires an Advanced Analytics platform to provide improved visibility of 13

grid conditions by utilizing EMS and PMU data. CSAT will use this data to display a graphical, visual 14

representation of the transmission grid in its current state. Operators can overlay circuit information with 15

other key data elements such as asset location, weather, traffic, and fire. These capabilities will 16

collectively give the operators early warning of developing events or impending faults to enable a 17

proactive mitigation strategy and faster, more targeted responses to those events. 18

This proposed solution will follow the recommendations outlined in the Federal 19

Energy Regulatory Commission (FERC)/North American Reliability Council (NERC) report on the 20

2011 San Diego blackout. The report concluded that improved real-time situational awareness would 21

have allowed the system operators both to take proactive action and to take timely restoration measures 22

to operate the system in a secure state without affecting millions of customers on a hot summer day, 23

potentially endangering public safety.158 Implementing the CSAT solution will significantly improve 24

SCE grid dispatcher situational awareness of fast moving abnormalities in the electric grid and will 25

provide targeted mitigation procedures to combat grid instability conditions. 26

157 Stabilization is a common IT process to monitor initial operations of new systems/equipment and optimize

their use in the production environment. 158 Refer to WP SCE-04, Vol. 2 Bk B pp. 194-349.

112

SCE continues to procure renewable power to meet the needs of our customers 1

and make progress toward State policy goals.159 As this happens, SCE’s grid control centers and 2

personnel must be equipped with upgraded tools to enable them to respond effectively to challenges 3

posed by higher penetration of renewables on our grid. Renewable generation sources produce 4

electricity intermittently, not on a predetermined schedule as is the case with more traditional power 5

sources. Both solar and wind power are accompanied by similar variability and reliability challenges. 6

This variability in generation from renewable sources makes it difficult to accurately forecast the 7

amount of expected output, which is a new challenge for Grid Operators who currently rely on 8

scheduled and accurate projections of generation output to maintain the grid in a balanced state. Other 9

challenges related to renewable generation include power quality, contingency planning, and predictive 10

analytics. These factors combine to increase the complexity of operating the grid and drive the need for 11

accurate tools to equip Grid operators with actionable information in a timely manner, improving 12

Situational Awareness in the Grid Control Center. 13

(1) Benefits 14

The CSAT project will provide benefits in three major areas listed below: 15

• Avoidance of Major Outages: Certain fast moving system disturbances 16

and resulting voltage instability could cause outages over a wide area 17

before mitigation actions can be taken. With real-time situational 18

awareness through PMU data and advance analytics, such disturbances 19

could be responded to and the impacts reduced. 20

• Improved Utilization of select Stability Limited Transmission Paths: 21

The analytics will enhance transmission models, specifically the 22

refinement of the dynamic models, to increase power load while still 23

meeting safety and operational requirements. These changes must be 24

coordinated and agreed to by the CAISO & Peak RC.160 25

159 State policy goals related to increasing the percentage of power deliveries from renewable resources can be

found in legislation such as Assembly Bill 32 and Senate Bill 350. 160 PEAK RC is Peak Reliability Coordinator provides real time monitoring of the Western Interconnection.

113

• Energy Procurement Benefits: SCE has conducted a study161 to 1

forecast benefits from increased utilization of transmission line 2

capacity, providing a potential opportunity to reduce energy 3

procurement costs. An average of $1.44 million and $2.36 million 4

annual savings on procurement cost of energy are estimated for a 5% 5

and 10% transmission capacity increase scenario, respectively. 6


Phase 1: CSAT – Real Time Monitoring and Analytics 8

In this phase, a new analytic platform consisting of the additional hardware and 9

software needed to run the application will be implemented. The system will integrate with SCE’s 10

Geographic Information System (GIS) and Energy Management System (EMS) for PMU data, and 11

provide the visualization capability to monitor real-time grid dynamics, such as phase angles, 12

oscillations, damping, and intelligent alarms. This phase will provide the foundational capabilities for 13

real-time monitoring. 14

Phase 2: CSAT – Enhanced Multi-layer Displays 15

In Phase 2, multi-layer displays will be implemented to enable effective 16

presentation of large amounts of information from various sources and applications, such as geographic 17

layers, electric circuit data, environment & weather data, and synchronized phasor data. The system can 18

diagnose and assess the system events and stressed conditions, plan actions, create preventative analytics 19

using the enhanced state, produce analytic compliance reports, and interface with multiple real-time 20

feeds from multiple sources.162 21

Phase 3: CSAT – Intelligent Displays, Events and Alarms for post-event analytics 22

This phase will provide the Grid Operators with trend displays, adding the 23

capability to view collected data plotted against time (e.g., the operator could observe a downward trend 24

in voltage that could lead to system instability). Operators can also play back and perform post-event 25

analysis. In addition, the CSAT system will enhance the prioritization criteria applied to incoming 26

alarms, improving the operator’s ability to separate and focus on high-priority alarms. The system will 27

display historical records and events using geospatial maps, present alarms on the geographical map, 28

161 Refer to WP SCE-04, Vol. 2 Bk B pp. 350-353. 162 Id.

114

assign priorities, and record change of priority. These features will benefit Grid Operators by providing 1

early warning of risks posed by environmental factors such as temperature, wind, wildfires, and changes 2

in renewable generation levels due to weather changes, etc. 3

The preferred solution is to purchase a COTS application to deploy the 4

capabilities mentioned above. The COTS application will be capable of integrating with SCE’s Grid 5

Analytics platform, which will enable seamless access to data from applications that directly monitor 6

and/or control the grid, as well as from other enterprise applications such as the GIS or services that 7

deliver environmental data. The integration of data from various sources will be implemented using a 8

data lake technology aligning with SCE’s analytical platform. 9

In our 2015 GRC, we forecast $13.1 million for this project. We now forecast 10

total project costs to be $21.8 million.163 The increase in cost is due to additional scope items (explained 11

above), such as integration with other enterprise data sets (e.g., geographical data, environmental and 12

weather data), post-event analysis, and intelligent alarms and events. The capital forecast for this project 13






Alternative 1: SCE considered building a customized solution on its 19

standardized data analytics platform. We did not pursue this option since the subject matter expertise 20

required for this endeavor is scarce and expensive in the industry. There are COTS applications 21

available today that will meet the business needs with some additional enhancements and integration. In 22

SCE’s experience, long-term support and maintenance of a custom application will generally cost more 23

than vendor costs to support a licensed COTS application. 24


115

15. Centralized Remedial Action Scheme (CRAS) Project 1

Table V-37 Centralized Remedial Action Scheme164


D.15-11-021 partially adopted the Centralized Remedial Action Scheme (CRAS) project. 2

SCE implemented CRAS in April 2016. The CRAS project included CRAS application development, 3

implementation of central control telecommunication infrastructure, and deployment of two RASs 4

(Remedial Action Schemes, defined below) to validate full capabilities of CRAS on SCE’s transmission 5

grid. As part of the final testing and implementation of CRAS, we have been able to observe that the 6

inability of GOOSE165 messages to be routable validated the need to upgrade the communication 7

capability of the architecture. This upgrade is discussed in the subsequent RGOOSE Project request.166 8

Remedial Action Schemes (RASs) are designed to detect predetermined system 9

conditions on the transmission grid and automatically take specific corrective actions such as tripping 10

generation or shedding load. When problematic system conditions are detected via RAS monitoring 11

relays, the RAS system arms and prepares to take protective action. This is our protection against 12

cascading outages and similar system conditions. If system conditions worsen, system load is shed (in 13

cases where load exceeds the remaining transmission capacity resulting in overloads or voltage collapse) 14

or generation is tripped (in cases where generation exceeds transmission capacity). This isolates the 15

system emergency, and prevents it from causing widespread services outages. 16

164 Refer to WP SCE-04, Vol. 2 Bk C pp. 3-91. 165 GOOSE refers to Generic Object Oriented Substation Events. 166 RGOOSE Project is a network upgrade project that improves the accuracy, failover capability, and

traceability of CRAS communications by upgrading the communications from current protocol (called GOOSE) to RGOOSE (Routable GOOSE). Please see the RGOOSE Project, presented below, for more information.

Forecast

CIT-00-SD-PM-000102 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 0.34 9.48 15.88 8.12 4.82 9.54 0.95 - - - - 49.13 2015 GRC Authorized* 0.34 9.48 15.88 8.12 33.82 2015 GRC - Original Request 0.34 9.48 15.88 13.92 9.77 - - - 49.39

* In D.15-11-021, the Commission adopted 2013 recorded expenditures and allowed SCE to reapply for capital expenditures in later years.

Recorded

116

As an example, a RAS might sense an overload on a transmission line it monitors. Based 1

on that reading, the RAS will send a signal to a generator, taking the generator offline to avoid 2

overloading lines and damaging equipment. Most RASs are far more complex than this simple example, 3

with many monitoring points and numerous conditions under which an action to either curtail generator 4

output, drop generation, or shed load would be initiated. Although technically complex, RAS systems 5

are significantly less expensive than constructing redundant transmission lines. Also, RASs can be 6

implemented much faster and avoid the extensive licensing processes required for new transmission 7

lines. RAS installations are approved by the Western Electricity Coordinating Council’s (WECC’s) RAS 8

Reliability Subcommittee (RASRS).167 9


In the 2015 General Rate Case, SCE proposed its CRAS (Centralized Remedial 11

Action Scheme) project at a cost of $49.392 million. In D.15-11-021, the Commission partially 12

approved the CRAS funding request by authorizing expenditures through 2013.While the Commission 13

Decision cited CRAS’s “intuitive appeal,”168 it disallowed recovery of costs for 2014 and 2015. In its 14

Decision, the Commission allowed SCE to reapply for the denied capital expenditures in our next GRC, 15

if we provided a detailed cost-benefit analysis in support of that request. We are providing that analysis 16

in this filing. The balance of unrecovered amounts between 2014 and 2016 is $15.31 million. 17

Since SCE’s transmission system is networked, we must protect against a single 18

transmission line outage resulting in a cascading blackout and wide-scale interruption of service. Our 19

transmission system network connects large, remotely located generators to major load centers and 20

connects our system to neighboring utilities. The reliability of the grid must be preserved as new 21

generation comes online. One factor we consider when a new generator interconnects is how the power 22

from that generator will flow under normal and contingency conditions. 23

167 “The purpose of the RASRS is to review the reliability aspects of existing and planned Remedial Action

Schemes (RAS) and to enhance grid performance within the Western Interconnection by providing a uniform review process.” https://www.wecc.biz/OC/Pages/RASRS.aspx

168 See D.15-11-021, pp. 43-44 (“[T]he intuitive appeal of the CRAS benefits that SCE describe are strong and the outcome of any effort to quantify them at this time may be primarily driven by preliminary assumptions (number of interconnections, policies on economic curtailment, etc.). As a matter of policy, this Commission supports a future with renewable generation resources operating efficiently on the grid and seeks opportunities to improve grid operations with respect to such resources. CRAS appears to be such an opportunity, and may be cost-effective in some scenarios. . . .”).

117

When new generators request interconnection and the existing transmission 1

system is not capable of handling all required contingencies, SCE has two choices. We can either build 2

redundant transmission lines or install a transmission system RAS. A redundant transmission line 3

provides the alternate pathway for energy during an outage and prevents other networked lines from 4

overloading. On the other hand, RASs work by shedding generation and/or load when a load flow 5

congestion condition occurs that risks system reliability or stability. This reduces power flows on the 6

system and prevents overloads that could lead to cascading outages. 7

Building many redundant transmission lines takes years to license and construct 8

and can be very expensive. These lines can cost as much as $9.5 million per mile and run dozens of 9

miles or more. SCE prudently pursues the construction of transmission line in some situations, but we 10

also rely on RASs to protect system reliability. To date, SCE has implemented 17 RASs across its 11

service territory. These systems are designed to automatically disconnect generators, load, or both under 12

certain identified system conditions. Prior to implementing CRAS, each RAS was designed as a 13

separate, self-contained system (e.g., a stand-alone system), including all of the relays, instrumentation, 14

and other equipment necessary for the RAS to function as designed. 15

SCE’s existing RASs are field-based (“stand-alone”) and have logic controllers in 16

the substations. CRAS, in contrast, has centralized decision-making and trip settings (“analytics”) 17

running on high-speed servers that are considerably more powerful than the field-based logic processors 18

in stand-alone RASs. Unlike a stand-alone RAS logic processor, CRAS is scalable by using additional 19

high-speed servers. This is important because the field-based logic processors have hard constraints 20

inherent in their design that limit the level of RAS complexity that can be accommodated. In contrast, 21

SCE designed and tested the initial hardware capacity of the CRAS system to be scalable to meet five-22

year “worst case” expected growth (30 RASs). SCE cannot envision a scenario where CRAS analytics 23

processing capacity would be exceeded by RAS data demands because CRAS blade servers are scalable 24

in processing power. SCE would either add additional blade servers, or upgrade existing servers, or 25

both. 26


The increasing RAS complexity that SCE described in its 2015 GRC application 28

occurred sooner than anticipated. The interconnect tariff requires four RASs in 2018-2020. These RASs 29

have a high complexity level. That complexity is composed of a higher number of generators, 30

118

substations, and lines to protect and, therefore, more contingency conditions to evaluate and on which to 1

take action. 2

The best measure of RAS capacity and complexity combines these factors in what 3

are called “arming points.” Arming points are the specific thresholds where a RAS will take action for a 4

particular generator as it evaluates a specific contingency (problematic system condition) related to that 5

generator or load. A generator might have one or more arming points based on the number and types of 6

contingencies involved. These arming points are defined by flow on transmission lines, in combination 7

with generation output that require mitigation if critical contingencies occur. Regarding specific arming 8

points for a generator, the contingency involved (such as a line being overloaded) might also apply to 9

other generators with other arming points. Arming points requirements, because they relate numbers of 10

contingencies and the number of generators, are a reasonable measure of a stand-alone RAS’s capacity. 11

Current stand-alone RAS logic capacity, without CRAS, is limited to 48 arming points due to limitations 12

in the relay hardware and firmware. 13

The technical problem that SCE faces is the complex stand-alone RASs needed in 14

the 2018-2020 timeframe would run out of capacity to add generators. In other words, a stand-alone 15

RAS would run out of available arming points. For example, one of these RASs (Northern Area RAS, a 16

new RAS) is significantly complex, requiring an estimated 76 arming points total to accommodate 17

Queue Cluster 8 (QC-8) or 124 arming points total when factoring in Queue Cluster 9 (QC-9).169 18

Northern area RAS cannot be accommodated as a stand-alone RAS.170 This is supported by SCE’s 19

planning and protection study. Therefore Northern RAS must be served by CRAS. The other alternative 20

to CRAS for the Northern Area RAS is building new transmission lines. This would be prohibitively 21

expensive,171 and the licensing and construction timeline would cause significant delays to two large 22

queue clusters in process (QC-8 and QC-9). Alternative design considerations of trying to “dumb down” 23

169 These queue clusters are groupings of projects in the interconnection queue that fall into similar geographic

areas and timeframes as describes further in the Generation Queue Completion section. 170 See testimony section Generation Queue Completion for additional details, which shows that most future

generation projects come to completion. 171 For more information on alternatives, refer to WP SCE-04, Vol. 2 Bk C pp. 9-56 and refer to WP SCE-04,

Vol. 2 Bk C pp. 57-71 (Northern Area Transmission tab), which shows transmission build timeline). In addition, the cost study compares CRAS for Northern Area versus a hypothetical stand-alone RAS for comparison purposes.

119

a stand-alone RAS approach (e.g., compress the required logic tables) have many undesirable effects, as 1

explained below. 2

In the past, SCE has compressed the logic tables for a stand-alone RAS (as 3

described in the 2015 GRC CRAS testimony), using Devers stand-alone RAS as an example.172 4

“Compression” is the grouping together of multiple contingencies on each logic step that would result in 5

sufficient corrective action. SCE considered using this approach for Northern Area, but there are two 6

potential problems: 7

1. Combining contingencies and/or generation would result in consistently 8

tripping more generation than needed for mitigation,173 which could expose the system to unwarranted 9

risks and potential incremental costs. Over-tripping uses more system spinning reserves than necessary, 10

which could cause the system to operate closer to stability limits. 11

2. Tripping more generation than needed for mitigation has impacts on the 12

generators. The more interruptions experienced by a generator, the more wear and tear occurs on the 13

mechanical and electrical systems of the generator; interruptions also cause an overall decrease in the 14

generator’s power production and associated revenues. 15

CRAS is needed to mitigate the potential issues caused by the compression of 16

contingencies/logic steps using a traditional stand-alone RAS. CRAS is designed to allow for 17

significantly more arming points, thereby permitting improved corrective action options that are tailored 18

to the specific needs of the grid, and avoiding over-tripping generation that is unnecessary. As 19

demonstrated in the workpaper entitled “Northern Area RAS Arming Points Comparison,” CRAS would 20

provide up to 189 arming points compared to the 45 provided by traditional RAS, providing a significant 21

increase in operational flexibility 22

(1) Analysis of additional complex RASs 23

Arming points requirements have been increasing significantly in 24

renewables-rich areas, as shown in the Figure V-7 below.174 25

172 See SCE-03, Vol. 2 of SCE’s 2015 GRC Application. 173 Refer to WP SCE-04, Vol. 2 Bk C pp. 72-74. 174 Refer to WP SCE-04, Vol. 2 Bk C pp. 75-78 and WP SCE-04, Vol. 2 Bk C pp. 79-80.

120

Figure V-7 Arming Points (Current & Future)

In addition, the diagram below Figure V-8 “RASs Nearing or Exceeding 1

Capacity Limits” provides a schedule of upcoming RASs, including Northern Area RAS in 2018. These 2

are the upcoming complex RASs in renewables-rich areas that are driving the need for CRAS.175 3


121

Figure V-8 RASs Nearing or Exceeding Capacity Limits

As a further illustration of the need for CRAS, please refer to Figure V-9 1

below on Northern Area RAS, which shows the numerous generators in queue. SCE is seeing large 2

queue numbers in renewables-rich areas, with still more expected as a result of the State’s 50% 3

renewables mandate by year 2030. 4

The challenge is more than just a RAS being complex at a certain point in 5

time. RASs in renewables-rich areas have many generators in queue, and this means a high rate of 6

change to the RAS. This has two undesirable effects concerning stand-alone RASs: (1) Any attempt to 7

increase capacity, such as by changing the relay CPUs (central processing units) and firmware, merely 8

creates another hard limit in arming points capacity that is likely to be exceeded in the future, and 9

(2) Changes to a stand-alone RAS are much more expensive than changes to a RAS in CRAS due to the 10

ease of making programming changes in CRAS centrally as well as considerably more streamlined test 11

procedures in the field to validate the RAS functionality. The CRAS Cost Study Overview and 12

Reference176 has a detailed explanation of the differences in testing methods and levels of testing 13

automation available in CRAS versus a stand-alone RAS which drive large differences in required labor 14

between the two approaches. 15


122

Figure V-9 177 Northern Area RAS Area Diagram with Generators in Queue

Regarding the method used to tally Northern Area arming points, when 1

QC-8 cluster additions are accounted for, 76 arming points are estimated for Northern Area. This is the 2

trigger for realizing we must use CRAS, as a stand-alone approach will not work. When we look further 3

out and add QC-9 cluster, total arming points for Northern Area climbs to 124 arming points, which far 4

exceeds the current maximum limit of 48 arming points for traditional stand-alone RAS, as well as the 5

66 arming points total offered by a stand-alone RAS capacity upgrade.178 6

(2) Generation Queue Completion 7

In the last two SCE GRC decisions, the Commission raised questions 8

about the percentage of queue projects that actually reach completion. A recent study on queue data 9

performed by SCE using CAISO queue data shows that a large majority of these projects do in fact 10

complete, and that the rate of completion is generally increasing over time. Table V-38 below shows 11

available CAISO queue data. As the numbers for individual years indicate, more projects can come on-12

177 Refer to WP SCE-04, Vol. 2 Bk C pp. 86-87. 178 Refer to WP SCE-04, Vol. 2 Bk C pp. 88-89.

123

line in a particular year because prior projects have slipped from their original proposed operating dates 1

to subsequent years. Also, it is not uncommon for more projects to come on-line in any given year 2

compared to what was expected at the beginning of a queue cluster study cycle. CAISO’s generator 3

interconnection study process is ongoing, with each queue cluster taking up about two and one-third 4

years from inception to completion. In any given year, there are typically two clusters that overlap each 5

other in a staggered manner. For instance, Queue Cluster 8 (QC-8) is in the Phase II study stage 6

concurrently with Queue Cluster 9’s Phase II study, with various reassessments in progress as well. 7

Detailed information on the application and study processes, timing, posting requirements, etc., can be 8

found at the CAISO link.179 9

Table V-38 Queue Completion Table

As shown in the Queue Completion Table, from 2006 to 2016, there were 10

41 queue applications. Thirty of these applications actually went into operation, which is an overall 11

completion rate of 73% over 11 years. The completion percentage has also increased significantly since 12

the CRAS project was launched, which is why the growth in complexity of RASs was greater in 13

magnitude and occurred sooner than originally expected. SCE must prepare to interconnect most of the 14

179 http://www.caiso.com/planning/Pages/GeneratorInterconnection/InterconnectionStudy/Default.aspx

124

generation that seeks interconnection. With the limitations inherent in stand-alone RASs, SCE cannot 1

rely upon local RAS control to achieve the interconnection requirements needed by the currently active 2

queue clusters.180 3

(3) Benefits 4

CRAS offers an easier, quicker, and less expensive solution than stand-5

alone RASs.181 Two key attributes that CRAS provides to achieve these advantages are a standardized 6

approach and automated test capabilities. These powerful features were designed to replace the 7

cumbersome, one-off customization needed for stand-alone RASs that increase the stand-alone RAS 8

costs at every stage—from design through installation, testing, and completion of commissioning. For 9

example, there are only three basic types of CRAS relays with identical settings for each type: 10

monitoring relays, customer-side mitigation relays, and utility-side mitigation relays. Stand-alone RAS 11

relays, in contrast, are individually customized. This customization drives increases in labor in all phases 12

of work, placing greater demand on SCE resources. 13

With detailed cost data for (1) the CRAS Project in hand, (2) the 14

underlying stand-alone RASs before they were brought into CRAS, and (3) many changes to stand-alone 15

RASs since 2013, SCE now has the data available to perform a more detailed cost-benefit comparison of 16

CRAS to stand-alone RASs. The cost-effectiveness for CRAS can be demonstrated for all new RASs 17

and certain types of existing RASs planned to be brought into CRAS. These RASs are existing complex 18

RASs in renewable-rich areas (e.g., Colorado River RAS, Whirlwind RAS, and Mojave Desert RAS) 19

that are running out of logic capacity, as explained above. 20

SCE considered a stand-alone RAS relay capacity upgrade to increase the 21

approximate number of arming points from 48 to 66.182 This would still leave the drawback of a hard 22

limit in arming points in place in the future for the stand-alone RASs, albeit a higher hard limit. SCE’s 23

cost study shows that such a capacity upgrade is prohibitively expensive due to high implementation and 24

change costs (as future generators will be added each year). CRAS addresses the expensive nature of a 25

stand-alone RAS installation through the following cost efficiencies: 26

• Reducing labor for each round of test. 27

180 Refer to WP SCE-04, Vol. 2 Bk C pp. 90-91. 181 Refer to WP SCE-04, Vol. 2 Bk C pp. 9-56 and WP SCE-04, Vol. 2 Bk C pp. 57-71. 182 Refer to WP SCE-04, Vol. 2 Bk C pp. 72-74.

125

• Reducing field crew requirements by eliminating the need for physical 1

inputs from field test sets in the final and most time-consuming phase 2

of testing (end-to-end field testing). 3

• Reducing the rounds of tests triggered by different design and logic 4

capabilities. 5

The complex RASs in generation growth areas brought into CRAS have 6

long test plans in terms of the number of test steps needed to validate the RAS’s functionality (250 – 7

1,000 or more lines of code). Stand-alone RASs and CRAS have very different test methodologies. 8

CRAS has automated test capabilities, which leads to dramatic differences in field labor requirements 9

for the same basic type of RAS validation. 10

The test methodology for a stand-alone RAS involves providing physical 11

inputs (currents and voltages) to field relays from test sets in the substations. Each test set typically can 12

only be hooked up to four relays at a time, depending on the scenario being tested. Frequently during the 13

lengthy test plans (250 – 1,000 or more lines of test steps) the test sets must be moved and connected to 14

different field relays. The substation environment is hazardous, making fast changes of temporary test 15

lead wiring unsafe. In summary, stand-alone RAS testing is cumbersome and slow, using significant 16

labor to validate the RAS. 17

By contrast, CRAS first validates physical inputs during the individual 18

testing that all relays must have, to make sure the relay is correctly sending and receiving data, and then 19

uses virtual inputs and automated test scenarios that can be loaded and activated centrally—a much 20

faster process than using the test sets with physical inputs as the stand-alone RASs do. 21

An additional benefit CRAS provides is that it eliminates the need to 22

repetitively enter physical inputs, which reduces the number of crews required per substation during 23

field end-to-end testing. In the end-to-end phase of testing, eliminating this need for physical inputs 24

reduces or eliminates the need for test sets, which are limited to about four relay connections per test 25

scenario. This in turn reduces the number of test technician hours needed, and represents a significant 26

reduction in required labor as RASs get more complex with larger test plans (e.g., more scenarios and 27

more line items in the test plan). 28

In addition, CRAS is more efficient in its test methodology and flexible in 29

its logic. As a result, we can consolidate designs and redesigns as generators are added, in order to 30

minimize the number of rounds of field end-to-end testing compared to stand-alone RAS testing. This 31

126

means that labor for CRAS testing is much more efficiently used, with fewer test hours per round of test 1

and significantly fewer rounds of end-to-end testing per year for the complex RASs ahead. In qualitative 2

terms, CRAS’s more efficient testing approach can cut times for scenarios (groups of test plan lines in a 3

test plan) from “hours” to “minutes” and time for a complete round of end-to-end testing in a complex 4

RAS from “weeks” to “days.”183 5

An additional CRAS benefit falls more into the area of risk mitigation than 6

direct cost avoidance: As part of the logic testing that SCE does centrally for CRAS logic validation 7

(e.g., the logic and algorithms present in the CRAS central controller servers), CRAS can centrally 8

execute more tests to validate logic, detect failures, and monitor communication volume and speed. 9

These additional tests, especially in the area of additional logic validation, would be prohibitively 10

expensive to perform for a stand-alone RAS because of the physical test set requirements described 11

above. The reason is that additional logic validation involves testing a range of increasingly unlikely 12

combinations of system conditions that would be high impact if they occur. This additional logic 13

validation is practical to test when high-speed CRAS automated testing is available. It is not practical 14

when stand-alone RAS test methods are used because field test plans would increase to thousands or 15

tens of thousands of lines. For a stand-alone RAS, this would result in a large increase in labor that is 16

beyond field crew availability constraints. In addition, it is impractical to schedule outages (with CAISO 17

and the generator customers involved) of the required length to do such additional testing for stand-18

alone RASs. 19

To clarify SCE’s plans to use CRAS going forward, not all existing stand-20

alone RASs will be converted into CRAS. We do not expect that all stand-alone RASs will reach the 21

threshold values of complexity or number of generators as the RASs in the cost study. The majority of 22

SCE’s existing RASs generally do not meet these criteria for conversion because of the conversion costs 23

to CRAS. It is the RASs in high renewables growth areas that typically meet the criteria for inclusion 24

within CRAS. Any RAS with arming points growth could meet criteria for inclusion even if their current 25

level of complexity is not that high because costs to accommodate growth in number of generators or 26

additional transformer banks are very high for stand-alone RASs and moderate with CRAS. SCE’s cost 27

study also confirms the benefits of deploying all new RASs into CRAS. 28


127

i. CRAS Cost Avoidance Summary 1

Table V-39 below shows the sum of expected cost avoidances for 2

CRAS for years 2018 – 2030 (in nominal dollars) in the more detailed cost study. A new transmission 3

build for Northern Area, as the only feasible alternative to CRAS, is a significant cost-avoidance driver. 4

In addition, the table contrasts stand-alone RAS implementation and change costs versus the 5

significantly lower implementation and change costs for CRAS. Designing and implementing stand-6

alone RASs is significantly more expensive than CRAS. Much of the difference is caused by high end-7

to-end field crew testing requirements for stand-alone RASs. These field testing requirements are much 8

more streamlined as explained above because a different test methodology is feasible and CRAS has 9

automated testing capabilities after the relays are given certain initial testing.184 10

Regarding Whirlwind RAS, at first glance it would appear that 11

there is lower than normal cost avoidance versus CRAS. This is due to the unusually high number of 12

existing generators served by the Whirlwind RAS. SCE tariffs require SCE to pay for CRAS change 13

costs in customer facilities. These costs are factored into the results of the cost study. 14

ii. CRAS and RGOOSE Cost Recovery 15

RGOOSE, as explained further in the RGOOSE section, is a 16

network communications protocol upgrade that serves as an enabler for further CRAS deployment. As 17

shown in SCE’s cost studies, CRAS appears to be more cost-efficient than alternatives. Therefore CRAS 18

cost avoidances (covered in detail in this CRAS section and CRAS Cost Study work papers) should 19

apply to both CRAS remaining unrecovered project costs and RGOOSE project costs. 20


128

Table V-39 CRAS Expected Cost Avoidance

(in Nominal Dollars)


The CRAS project consisted of system deployment work, including procuring, 2

developing, and testing software applications for communications processing, grid monitoring, 3

performance self-monitoring, security monitoring, security monitoring and controls, and protection logic 4

algorithm management. The project scope also included procuring, configuring, testing, and installing 5

servers and racks for these applications. Project implementation included converting two RASs into 6

CRAS as part of the deploying and demonstrating CRAS. SCE tested these two RASs in parallel with 7

the existing stand-alone RAS’s, and they are currently operational as part of the CRAS system. 8

Telecommunications work included fiber modifications, microwave link modifications, new switches, 9

and new routers. Substation equipment and telecommunication equipment within the substation must be 10

physically and electronically protected from unauthorized access to comply with NERC/CIP standards. 11

SCE included the construction of these physical and electronic security perimeters as part of the project 12

deliverables. 13

SCE is estimating revised total CRAS project costs of $49.13 million from 2010 14

through 2016. In our 2015 GRC, SCE requested $49.39 million for total CRAS project costs. D.15-11-15

129

021 allowed recovery of recorded costs from 2010 through 2013, which totaled $33.82 million. This 1

leaves remaining unrecovered CRAS Project costs at $15.31 million.185 2


Alternative 1: SCE considered not implementing this “project” (in this 4

sense, by “project” we mean further deployment of CRAS, not the base CRAS project already 5

implemented). Instead, for the three RASs out of the four under consideration where a stand-alone RAS 6

capacity upgrade is possible, we considered increasing the capacity of stand-alone RAS relays instead of 7

converting the RASs into CRAS. This alternative is not recommended because it is cost-prohibitive and 8

merely sets another hard arming-points limit that will be exceeded in the future, eventually causing the 9

same capacity shortfall. In addition, a critical deficiency of this alternative is that Northern Area RAS 10

(the remaining RAS out of the four under consideration) cannot be accommodated as a stand-alone RAS 11

because it would not meet grid protection requirements186 even if there is an increase in stand-alone 12

RAS relay capacity (see Alternative 2). Alternative 1 provides no long-term solution to arming-point 13

capacity and is not cost-efficient. 14

Alternative 2: SCE considered not implementing this “project” (in this 15

sense, by “project” we mean further deployment of CRAS, beyond the base CRAS project already 16

implemented) and instead, build a transmission line to meet the Northern Area protection needs. This 17

alternative is not recommended due to costly infrastructure and lead time requirements that would 18

significantly impact the feasibility of QC-8 and QC-9 renewables generation projects. 19

185 Refer to WP SCE-04, Vol. 2 Bk C p. 8. 186 Refer to WP SCE-04, Vol. 2 Bk C pp. 88-89.

130

16. RGOOSE Project 1

Table V-40 RGOOSE Project187



The RGOOSE Project is a communications network upgrade project. It takes 3

advantage of the significant communication improvements offered by a new industry standard for 4

network communication and security.188 This industry standard improves the existing GOOSE (Generic 5

Object Oriented Substation Events) protocol used in substations to Routable GOOSE (RGOOSE). 6

The GOOSE protocol sends data packets (like small messages) in broadcast 7

fashion, much like a radio network. The packets go to all network points and all devices “hear” the 8

messages whether they need to or not. This broadcast approach puts a substantial burden on a large 9

network.189 RGOOSE significantly improves performance by making the protocol “routable” by giving 10

each packet precise targeting to go only where it needs to go, to a specific device or set of devices. As 11

networks expand beyond the substation (such as a large network combining many substations and 12

control centers), this precise targeting is important for the capabilities of the system and for effective 13

troubleshooting. The RGOOSE communication protocol will first be used by SCE to meet the growing 14

data communications needs of CRAS project. As described in the following sections, RGOOSE is 15

required for CRAS moving forward to accommodate the high data volume and complexity of network 16

communications associated with upcoming RASs that contain significantly greater arming point 17

requirements. 18

187 Refer to WP SCE-04, Vol. 2 Bk C pp. 94-106. 188 Industry standard IEC 61850 90-5. Background information: https://www.pacw.org/no-

cache/issue/december_2013_issue/network_architecture/microgrids_integrating_renewable_energy_sources_res_to_improve_reliability/complete_article/1/print.html.


CIT-00-SD-PM-000243 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total

Recorded / Forecast - - - - - 4.50 5.90 - - - 10.40 Previous GRC Request - - - - - - - -

Recorded Forecast

131


The following diagram provides a simplified view of the CRAS system and 2

indicates the high flow of data at all points in the communication structure. For example, monitoring 3

relays send information about the status of transmission lines to the central controllers, and mitigation 4

relays receive instructions about the generation tripping to be executed through opening circuit breakers 5

at the generation plant interconnection. 6

Figure V-10 CRAS Simplified System Diagram

In Figure V-10, “CRAS Simplified System Diagram,” blue lines indicate high-7

speed diverse path communication links with switches and routers. A and B side monitoring and 8

mitigating data go to and from control centers. Per WECC requirements, RASs have A and B sides and 9

diverse paths for redundancy/reliability purposes. The small arrows represent a high volume of 10

messages. CRAS central controllers are servers in control centers that evaluate monitoring data and send 11

mitigation signals. Mitigation relays trip generation or shed load as needed to protect the transmission 12

grid (via circuit breakers not shown). 13

RGOOSE is the communication architecture that enables high speed and reliable 14

communication between the substation and control center. Its central value is the ability to target 15

messages to the specific devices that need them, providing greater accuracy and traceability for CRAS 16

communications, as well as adding communications failover capabilities that a GOOSE network cannot 17

provide. 18

Our experience during late phases of testing and since CRAS implementation has 19

validated the capacity of relay equipment to emit extremely high volumes of data, as this characteristic 20

is required to meet the control functions of CRAS. The high data traffic in the current GOOSE network 21

132

raises concerns that, due to the required GOOSE tunnel architecture, the communication equipment may 1

not be capable of handling the expected increase in transmitting all data packets reliably in the current 2

design.190 With the current GOOSE deployment, the system will not be able to satisfactorily handle the 3

increased data flow for the more complex RASs such as Colorado River, Northern Area, Mojave Desert, 4

and Whirlwind. 5

The only way to effectively use GOOSE protocol on a large network is to limit 6

where it can go. To accomplish this communication, tunnels are created on the network. These tunnels 7

force communication from substation A to go only to a limited number of devices (such as the central 8

controllers in the control center) but not get mixed with communication from substation B on the wider 9

network. Each substation requires multiple tunnels for GOOSE to be operating effectively. 10

Unfortunately, the settings for the communications devices to create the tunnels are one-off, customized, 11

and cumbersome to configure. The proliferation of these cumbersome tunnels as a GOOSE network 12

grows in size raises performance risks as more substations are added. 13

Two additional challenges exist with GOOSE: First, there is a concern that as 14

more relays are “talking” and the tunnel architecture is overwhelmed, critical data packets can be 15

dropped. This problem can occur when tunnels fail. Second, as additional tunnels are created to enable 16

more communications, diagnosing failures becomes extremely challenging. Each device in the network 17

is individually customized and is not addressable on an individual basis, making it considerably more 18

difficult to track and troubleshoot problems. To use an analogy, the monitoring relays speaking GOOSE 19

can be viewed as a group of radio stations emitting different radio programs. Each mitigation relay must 20

listen to all of the programs on a particular tunnel, but only operate when it hears a particular song. In 21

contrast, RGOOSE architecture is more akin to a telephone exchange where relays are called and given 22

instructions without every relay having to listen to the phone call targeted for only one relay. 23

With the more complex RAS requirements SCE now faces, RGOOSE is needed 24

so that the functional requirements of CRAS can be fulfilled and the full benefits of CRAS are realized. 25

The first system to benefit from the use of the RGOOSE communication protocol will be CRAS. The 26

description, business need, and benefits of the CRAS solution are provided in the section above. To 27

move beyond the initial substation deployment of CRAS and support the full deployment of this 28


133

solution, we need to use RGOOSE. We need to adopt this more advanced protocol because of 1

significantly greater number of arming points191 in upcoming RASs. 2

The most significant arming point challenge SCE faces is with the Northern Area 3

RAS, which requires 76 arming points in the near term and 124 in the longer term, as explained further 4

in the CRAS project presented earlier. Each arming point drives continuous messaging traffic on the 5

network about the conditions affecting that arming point, such as electric current flow or the status of a 6

transmission line. This data volume requires the certainty of message delivery as well as the rapid 7

diagnostics of communication failure. We can obtain these capabilities by deploying RGOOSE on the 8

CRAS network for future RAS deployments. 9

The RGOOSE Project has been developed to take advantage of the now-available 10

standard (IEC 61850 90-5) to overcome the limitations inherent in GOOSE. The RGOOSE standard was 11

not available when CRAS project was launched in 2011; it was necessary to use the protocol available at 12

the time (GOOSE) to design, implement, and test the central controllers for CRAS. The industry has 13

been developing the RGOOSE protocol for several years due to increasing data needs on the electric 14

grid. SCE is able to upgrade to RGOOSE as it is now technically and commercially feasible to 15

implement. In the meantime, a great increase in RAS complexity occurred as the rate of generation 16

queue project completion increased to substantially greater levels compared to when the CRAS project 17

was first launched. This increased complexity leads to the need for the RGOOSE network upgrade 18

shortly after CRAS project implementation. 19

(1) Benefits: 20

The primary benefits of RGOOSE are: 21

• More accurate, stable, and secure high-speed communication on the 22

wide area network, regardless of the volume of communication. This 23

reduces the risk of outages on the transmission grid. 24

191 As explained in the CRAS section, the increasing RAS complexity that SCE described in its 2015 GRC

Application is being experienced, with four RASs required in 2018-2019 that have a high complexity level. That complexity represents a higher number of generators, substations, and lines to protect—and therefore more contingency conditions to evaluate and take action on. The best measure of RAS capacity and complexity combines these factors in what are called “arming points.” Arming points are the specific thresholds where a RAS will arm for a particular generator as it evaluates a specific contingency (problematic system condition) related to that generator.

134

• Better capabilities in monitoring internal communications and 1

troubleshooting. This leads to quicker restoration if a communication 2

problem does occur. 3

Adopting RGOOSE brings the network capability up to match the 4

advanced capabilities of the CRAS platform central controllers and field relays. 5

b) Scope and Cost Forecast 6

Upgrading network gear and central controller software to the RGOOSE protocol 7

to prepare for upcoming RAS deployments, consists of four main areas: 8

1. Upgrading relays. General Electric Company will deliver RGOOSE-9

compatible test relays. This is a relatively minor cost component (General Electric incurred the design 10

and development costs to convert their relays to RGOOSE). 11

2. Upgrading to RGOOSE-capable networking gear (switches and routers for 12

test purposes), including communications network design costs. 13

3. Upgrading the CRAS central controller software to use RGOOSE, which 14

includes costs for designing and developing the upgrade. 15

4. Testing key aspects of the overall CRAS system using RGOOSE in Site 16

Acceptance Test - Cycle 1 to make sure the communications changes will succeed. See cost workpaper 17

for additional details. 18

The actual rollout of communications networking for a particular RAS and the 19

detailed testing of that particular RAS are not included in the RGOOSE project. Instead, these items are 20

covered as part of infrastructure work performed regardless of the conversion to RGOOSE. The network 21

gear will be at similar cost to the current CRAS network gear (or, communications gear for stand-alone 22

RASs). The testing for CRAS deployment for particular RASs will be much more comprehensive, 23

automated, and efficient than for stand-alone RASs and will therefore be highly cost-effective.192 24

SCE launched the RGOOSE project in June 2016, and it is expected to be 25

completed in Quarter 2 of 2017 (with completion of the first cycle of Site Acceptance Testing). SCE will 26

then deploy CRAS using RGOOSE for upcoming RAS infrastructure investments. The net present value 27

of the T&D capital cost avoidances more than offset the CRAS Project and RGOOSE Project costs, as 28

192 Refer to WP SCE-04, Vol. 2 Bk C pp. 9-56 and WP SCE-04, Vol. 2 Bk C pp. 57-71.

135

discussed in the CRAS Project section of the filing and associated work papers. RGOOSE is a key 1

enabler for further CRAS deployment, as described in this section. Therefore CRAS cost avoidances 2

(covered in CRAS section and CRAS Cost Study) should cover both CRAS remaining unrecovered 3

project costs and RGOOSE project costs. 4







Alternative 1: SCE considered not implementing this project. In other 11

words, we considered not implementing the RGOOSE protocol193 and not deploying CRAS further due 12

to the risks of using GOOSE protocol with significantly more complicated RASs. Instead, we could 13

implement stand-alone relay capacity increases on three of the four RASs (Colorado River, Mojave 14

Desert, and Whirlwind) and a new transmission build for Northern Area (i.e., the remaining RAS that is 15

too complex for a stand-alone RAS architecture). This combined alternative is not recommended, as 16

described above in the CRAS testimony. For three of the four RASs, this alternative does not provide a 17

long-term solution to capacity issues, and it is not cost efficient. For Northern Area RAS, the only other 18

alternative besides CRAS with RGOOSE is building out very expensive and redundant transmission. 19

Alternative 2: SCE considered not implementing the RGOOSE Project 20

and instead deploying CRAS with the existing GOOSE network approach. This alternative is not 21

recommended, as complex data volumes make it technically difficult and cumbersome to determine 22

where a failure occurs without using RGOOSE protocol. Though it was necessary to have a wide area 23

network approach to properly develop and test the CRAS platform, GOOSE is less stable than 24

RGOOSE, and its non-standard usage on the wide area network would be imprudent, given that a better 25

standardized approach is now available. SCE has seen a large increase in RAS and required arming 26

point complexity. This increased complexity drives higher data volume requirements and the need for 27

the stronger, more stable, and more precise RGOOSE communications network protocol. 28


136

17. Energy Management System (EMS) Refresh 1

Table V-41 Energy Management System Refresh194



The Energy Management System (EMS) is operated by the Grid Control Center 3

(GCC). The GCC monitors and controls the bulk power system 24/7, using SCE’s EMS, and its output is 4

displayed on a video wall that graphically depicts the status of SCE’s and neighboring utilities’ bulk 5

power systems. 6

This project will refresh the aging hardware and software components of the EMS 7

system to maintain system availability. In addition to a technical refresh of the EMS hardware and 8

software, this project will consolidate the EMS and Phasor systems. SCE’s Phasor system collects, 9

stores, and shares Phasor Measurement Units (PMU)195 data. Consolidating EMS and Phasor onto a 10

single platform will reduce hardware and software costs and simplify the maintenance of both systems. 11

It will also provide a unified operator view of system conditions, eliminating the need for two separate 12

but related systems. 13


The primary driver for the EMS Refresh Project is the aging EMS system, since 15

the vendor will not support the current version after 2017. The risk of hardware failure continues to rise 16

as the system ages. By the time of the refresh, the hardware will be 7 years old. Refreshing EMS will 17

allow SCE to maintain system reliability at 99.95% or greater and reduce the risk posed by components 18

reaching the end of life. System reliability of EMS is important to SCE Grid Operations because EMS 19

194 Refer to WP SCE-04, Vol. 2 Bk C pp. 107-115. 195 A Phasor Measurement Unit is a device installed in the field that monitors & reports electrical signal on the

grid.

CIT-00-SD-PM-000231 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 6.21 7.22 2.67 - - 16.10 Previous GRC Request - - - - - - - -

Recorded Forecast

137

system failures could impair the system operators’ ability to safely operate the grid. Besides the 1

reliability needs, the current EMS configuration is also nearing its maximum data storage and processing 2

capacity, and requiring expansion. 3

(1) Benefits: 4

In addition to the operational benefits of the technical refresh described 5

above, consolidating EMS and Phasor systems will eliminate the following: 6

• The need to maintain dual systems, providing savings on Compliance 7

and Operational support activities. 8

• The need to refresh each of the EMS and Phasor systems 9

independently, resulting in a cost avoidance of $7.0 million. 10


The EMS Refresh scope will replace aging computing hardware and network 12

equipment critical to maintaining communications with field devices. It will also enhance existing test 13

environments to provide more effective testing prior to deployment. In addition, the refresh project will 14

upgrade system and application software and expand user capacity to meet projected future growth. 15

The EMS and Phasor consolidation will migrate the separate instances of the GE 16

solution to a shared infrastructure. This consolidation will substantially reduce the total number of 17

components in the system, reducing maintenance and support costs. 18

This project was launched in June 2016 and is expected to complete in mid-2018. 19



components. SCE’s forecast for this project includes costs for SCE employees, supplemental workers, 22

and consultants, software and vendor costs, and hardware costs. See this project’s workpaper for the cost 23

breakdown information.196 24



modification. We did not pursue this option as the current version of EMS will not be supported by the 27

vendor, and it does not provide on-going savings realized from consolidation of EMS and Phasor. 28

196 Refer to WP SCE-04, Vol. 2 Bk C p. 115.

138


performing a refresh of EMS hardware and software now, but not consolidating the Phasor and EMS 2

systems. We did not pursue this option because conducting the EMS Refresh and Phasor Refresh 3

projects independently will lead to higher costs, both in terms of project and maintenance costs. This 4

alternative is estimated at $23.04 million as compared to the selected approach, estimated at $16.1 5

million. 6


performing an EMS hardware refresh without upgrading software and not consolidating the Phasor and 8

EMS systems. We did not pursue this option for the same reason as alternative two. In addition, this 9

option would not address the risk posed by third-party-software components also nearing end of life. 10

This solution is also expected to have costs higher costs ($20.5 million) compared to the selected option 11

($16.1 million). 12

18. Outage Management System (OMS) Refresh 13

Table V-42 Outage Management System197



D.15-11-021 adopted the Outage Management System (OMS) Refresh project as 15

part of SCE’s 2015 GRC. OMS is used by SCE’s Grid Operations division to monitor, identify, and 16

operate electrical network and associated assets during electrical system outages on SCE’s distribution 17

network. 18

We have made cost and schedule adjustments to the project since originally 19

proposed. The project was originally scheduled to begin in July 2014 and complete in June 2016. The 20

project start date was delayed to November 2014 and will now end in December 2017. The schedule 21


CIT-00-DM-DM-000050 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - 1.05 1.94 6.80 3.50 - - - 13.28 Previous GRC Request* - - - 1.60 5.10 3.10 - 9.80 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

139

changes were necessary to accommodate an extended pre-engineering phase due to scope increase and 1

related vendor scope re-definition. 2

The addition to scope from the previous filing can be categorized into two main 3

areas aligned with the release schedule of the Project. Those two areas, further described in the scope 4

and forecast section below, are: (1) a Transmission Substation-level (Trans-Sub) Connectivity Model; 5

and (2) additional OMS enhancements, including further integration with Smart Meter functionality. 6


A refresh of the OMS system is required to maintain system availability, usability, 8

and reliability, as the current version will no longer be supported by the vendor. This creates a 9

significant operating risk, as SCE’s Grid Operators cannot identify and/or respond to customer outages 10

effectively if OMS is not available. This refresh will also enhance system performance and the ability to 11

report and assist in mitigating customer outages during peak system use periods. 12

(1) Benefits 13

First, refreshing the OMS system will retain the vendor support required 14

for on-going operations and help deliver more reliable system operations and performance. The OMS 15

technical refresh will also improve availability and reliability of interfaces with other key SCE operating 16

systems that enable OMS to reduce system down time and meet business objectives. 17

Second, the current system has a stand-alone Distribution circuit model, 18

and it does not show the connections between the Distribution system and the Transmission system. This 19

lack of visibility to the Transmission network results in analysts having to do time-consuming manual 20

analyses to fully determine outage impacts. The availability of the Trans-Sub Connectivity model in 21

OMS will provide direct visibility to Transmission level outages, allowing better grouping of affected 22

distribution circuits, reducing manual analysis, and help streamline dispatch decision-making. The 23

Trans-Sub Connectivity model implementation in OMS is a building block toward establishing an end-24

to-end Grid Connectivity model.198 25

Third, we have identified a set of enhancements to the current OMS 26

functionality. Implementation of automated methods to prioritize outage incidents will lead to better 27

decision-making during storm response and help reduce outage response times. In addition, the 28

198 See Grid Connectivity Model in SCE-02, Vol. 10.

140

capability to seamlessly “ping” customer meters from OMS, both individually and by groups of meters, 1

to confirm an outage or restoration in near real-time will assist in timely decision-making by Grid 2

Operations. 3


The project will be executed in a three-phase approach: (1) upgrade the current 5

application (CGI) to its latest version, (2) implement enhancements, and (3) implement a Trans-Sub 6

Connectivity model. 7

The OMS System Refresh scope as defined in our 2015 GRC testimony included 8

the implementation of the vendor’s OMS version 6.5. This will be carried out in the first phase. 9

Capabilities available in the new OMS software include a web-based version of OMS for users other 10

than system operators to reduce any performance impact of their use on the primary system. Phase 2 will 11

implement (1) enhanced graphical functions including improved circuit and substation visualization that 12

will greatly benefit the end users, (2) a dynamic scoring model for improved outage prioritization, and 13

(3) improvements through closer integration of OMS with Smart Meters. Phase 3 will implement 14

changes to OMS and its interfacing systems to deploy the Trans-Sub Connectivity Model capability. 15

The project cost proposed in our 2015 GRC application was $9.8 million. The 16

revised project cost for this rate case request is $13.3 million199 to complete application development, 17

training development and delivery, and implementation. The increase of $3.5 million is attributed to (1) 18

an increase in scope and (2) an increase in vendor cost estimates due to a three-phase implementation as 19

opposed to a higher-risk, single-phase approach as proposed in the 2015 GRC. 20




supplemental workers, and consultants, as well as software and vendor costs. See this project’s 24



Alternative No. 1: SCE considered keeping the existing solution without 27

modification. We did not pursue this option because it would lead to significant risk to (1) system 28

199 Refer to WP SCE-04, Vol. 2 Bk C pp. 123.

141

availability as the product will be unsupported by the vendor, and (2) reliability and public safety while 1

SCE operates manually during times of OMS system outage. Any associated benefits related to 2

enhancements and the Trans-Sub Connectivity model would not be realized under this alternative. 3

Alternative No. 2: SCE considered enhancing the existing solution by 4

refreshing the current application to its latest version (i.e., technology upgrade only). We did not pursue 5

this option as it would prohibit SCE from realizing previously identified benefits from storm response 6

and Smart Meter integration, and any associated benefits related to enhancements and the Trans-Sub 7

Connectivity model. 8

19. Distribution Management System (DMS) Refresh 9

Table V-43 Distribution Management System200



D.15-11-021 adopted a revised forecast for the DMS project. DMS is the 11

centralized computing system used by SCE to gather data from various distribution automation devices 12

and facilitate automated operation and control of the distribution system. DMS is an essential 13

distribution system for operating our system and maintaining its reliability. More automated equipment 14

and devices with remote-control capability are added to the distribution system every year to test current 15

conditions, isolate problems, and restore service more quickly. 16

The DMS project was proposed to be implemented in two phases. Phase 1 was 17

implemented in 2012 and provided the following benefits: 18

• A custom-built distribution management system was replaced by a COTS system, 19

which provided full vendor support for defect resolution and future upgrades; 20


CIT-00-SD-PM-000140 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 7.74 6.23 5.51 5.62 1.73 5.09 1.50 - - - - 33.41 2015 GRC Authorized* 7.74 6.23 5.51 5.62 4.84 - - - 29.93 2015 GRC - Original Request 7.74 6.23 5.51 8.30 4.84 - - - 32.61 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized our 2014 request.

Recorded Forecast

142

• The ability to handle new automated devices added to the distribution network to 1

better analyze circuits, leading to better planning and remediation of system problems; and 2

• Improved capabilities of meeting security requirements for authentication of user-3

command and control data. 4

The Phase 2 schedule was revised due to technical challenges involved with 5

enhancing a COTS solution to meet operational requirements. These operational requirements were 6

associated with user interfaces, system alarm settings and notifications, and reliable and safe operation 7

of field devices. The COTS product required multiple iterations of enhancements and testing to meet 8

these operational requirements which resulted in a rescheduling of implementation and rollout activities. 9

Phase 2 was planned to be implemented in three releases: 10

• Release 1 was implemented in 2014 and deployed AVVC (Advanced 11

Voltage VAR control) capability; 12

• Release 2 was implemented in 2015 and deployed Operation Training 13

Simulation capability; and 14

• Release 3 will be implemented in 2016 and will deploy advanced 15

control and analysis capabilities. 16

Phase 2 will provide following benefits: 17

• AVVC (Advanced Voltage VAR control) is a method of Conservation 18

Voltage Reduction (CVR) where automated field capacitors on distribution circuits optimize 19

overall customer voltage and reduce energy use for distribution purposes. This advanced control 20

allows SCE to provide the required quality of service (i.e., voltage level) to its customers, but 21

with less purchased power.201 22

• An operator training-simulation feature for all switching center 23

operators, which provides a realistic behind-the-wheel training experience that tests the 24

operator’s capabilities to perform under a real-world stress situation; and 25

201 See SCE 2015 GRC, SCE-05, Vol. 2, Pt.2, Table IV-27 “Estimated DMS Avoided Purchase Power Benefits.”

143

• An advanced control and analysis capabilities allowing SCE to 1

implement self-healing security,202 which provides automated responses to system problems. 2


SCE has recorded $31.9 million through 2015 and expects to spend an additional 4

$1.5 million to complete the project in 2016. This total project cost of $33.4 million is above the 5

Commission’s 2010 – 2015 authorized expenditures of $29.9 million. This is primarily due to additional 6

testing needed to meet operational requirements. 7

In our 2015 GRC, SCE estimated that the total cost of completing the DMS 8

project would be $32.6 million, as reflected in Table V-43. SCE is requesting expenditures in this GRC 9

to complete the project at a total cost of $33.4 million. This increase relative to our original request and 10

our authorized amount from the GRC is due to two primary factors described below. 11

First, in D.15-11-021, the Commission authorized 2013 recorded costs for this 12

project. SCE underspent in 2013 relative to our requested amount by $2.7 million, primarily due to the 13

technical challenges in aligning the commercial product to SCE’s operational needs. These challenges 14

delayed a portion of the original scope of work for 2013 to future years. As this scope of work is integral 15

to DMS performance, SCE subsequently completed this work to enable the project to provide its 16

intended benefits. Since the scope that the Commission adopted for this project remains through this 17

request, the original forecast from the 2015 GRC is still the appropriate benchmark of costs required to 18

achieve the adopted project scope. 19

Second, the complexity of the DMS project required longer periods to test, 20

stabilize, and implement the functional and technical capabilities of the system. As the DMS is 21

responsible for core aspects of our electric system operations, this extended effort was critical to our 22

ability to introduce DMS upgrades in a phased approach that could handle the large volume and 23

diversity of data, while maintaining system and operational performance standards. 24




202 Self-healing is the ability of the system to detect that it is not operating correctly, and without human

intervention make the necessary adjustments to restore itself to normal operations. This feature will provide capability to detect faults and restore the grid to normal operations.

144


workpaper for the cost breakdown information.203 2

20. Grid Interconnection Processing Tool (GIPT) 3

Table V-44 Grid Interconnection Processing Tool204



The Grid Interconnection Processing Tool (GIPT) is a suite of integrated 5

workflow and project management software tools that will allow customers to more quickly and 6

efficiently connect electrical generation and load to the grid. This project will implement a software 7

solution to handle various types of interconnection requests such as Net Energy Metering (including 8

NEM 2.0), Rule 21 (Export and Non-export), Wholesale Access Distribution Tariff (WDAT), and 9

Transmission Operated Tariff (TOT). GIPT will speed up the interconnection project lifecycle, improve 10

the customer experience, and provide SCE with more timely and accurate interconnection data. The 11

GIPT will be a suite of software applications, composed of commercially available products and custom 12

built tools integrated together by a single workflow engine. The custom built tools will be a co-13

development project between SCE and a leading software integration vendor. 14


Regulatory drivers,205 customer choice, and technological advancement are 16

driving increased adoption of distributed energy resources (DERs) in SCE’s service territory and the 17

resulting need to efficiently process interconnection requests. We predict the current and forecasted 18

future volume of DER interconnection requests will be constrained by SCE’s existing tools and 19

processes. Customers can submit digital interconnection requests online for Net Energy Metering and 20

Rule 21 Non-export projects. Customers submit hard copy and/or soft copy interconnection requests 21

203 Refer to WP SCE-04, Vol. 2 Bk C p. 130. 204 Refer to WP SCE-04, Vol. 2 Bk C pp. 131-133. 205 See D.16-06-052, pg. 25, 32, 37.

Recorded Forecast

CIT-00-OP-NS-000520 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 0.46 6.90 6.32 - - 13.68 Previous GRC Request - - - - - - - -

145

through multiple mechanisms for the remaining tariffs and programs (WDAT, Rule 21 export, TOT, 1

Queue Cluster, and others). Forms filled out by hand cannot be automatically validated and require 2

manual verification and processing. This creates delays, as SCE personnel must manually substantiate 3

the technical details based on information that may lack data accuracy and consistency. 4

Grid interconnection data (such as technology type, size, location, status, 5

contracts, ownership, and study reports) are stored in a variety of disparate databases. System planners 6

and other technical stakeholders manually import generation data to perform time-consuming technical 7

studies. Given there are multiple interconnection request entry points, all interconnection projects are 8

managed with independent trackers maintained by different organizational units, and do not integrate 9

with the intake pathways. 10

These multiple disparate databases/trackers often contain conflicting records and 11

pose significant challenges with system planning and operations, and publishing system information 12

through external portals in the public domain. Customer satisfaction is affected by the inability to 13

quickly present customers with potential system constraints and associated interconnection costs. 14

Various entry points for generation and load interconnection requests can pose a communication gap 15

between the organizations receiving the requests and the organizations responsible for planning the 16

system (and those creating and managing the interconnection contracts), complicating the roles and 17

responsibilities of lifecycle project management. Finally, legacy payment mechanisms (e.g., check, wire 18

transfer) combined with manual work order creation can also cause delays throughout the 19

interconnection process. 20

146

Figure V-11 NEM Application Growth

2008-2016

Note: 2016 is a forecast. Prior years are actual

As shown in Figure V-11, SCE is experiencing an increased volume in 1

interconnection requests and greater diversity in technology types being deployed. These factors further 2

increase the level of complexity required to perform system analysis. If SCE does not improve the tools 3

that are available to handle and manage these requests, there may be interconnection delays. 4

Load interconnection activity largely influences SCE’s system planning and 5

forecasting. Today, customers typically submit a new load service request or an increase of existing load 6

service in hard copy or soft copy format through local planning offices. Phone calls, meetings, and email 7

are the primary means of communicating these requests to planning organizations, which are then stored 8

in planning tools to be included in system forecasting and capacity planning. Implementing GIPT will 9

help improve data quality and interconnection work process management across organizations. 10

Impacts to Interconnection Customers: 11

Generators and demand-side resources connect to SCE’s system through many 12

tariffs and programs. Some generators interconnect by mailing paper applications, while others can 13

access online forms. While SCE’s NEM program now offers the ability to complete an interconnection 14

request online, many other Rule 21 and wholesale generators interconnecting under SCE’s WDAT 15

147

submit applications in paper or paper scanned to email. Customers often sign up for distributed energy 1

resource programs online or submit paper applications to local planning authorities. SCE staff store and 2

process these applications in various unlinked databases. Depending on how those applications are 3

processed, an SCE system engineer evaluating the impact of an interconnection on the system will not 4

have access to the most up-to-date information about other pending or recently-completed 5

interconnections. 6

GIPT will serve as a single, uniform repository for all interconnections to SCE’s 7

grid. GIPT will enable all interconnection requests, contracts, and related information to be digitized and 8

available in a common repository. 9

Impacts to SCE’s internal processes: 10

To perform accurate and efficient system planning and operations, SCE personnel 11

will need a Grid Connectivity Model (GCM), described later in this volume, which depicts an accurate 12

representation of the system. The GIPT is the GCM’s primary source of DER technical and contractual 13

information. Absent the GIPT, the GCM will contain an incomplete set of DER technical and 14

contractual information and therefore cascade inaccuracies to dependent systems. 15

(1) Benefits 16

The goals of GIPT are to improve the overall customer experience 17

throughout the interconnection project lifecycle by reducing study costs, providing a higher certainty of 18

interconnection viability, and accelerating interconnection-request turn-around times. 19

The GIPT is a single, web-based user interface that allows customers to 20

submit interconnection requests for generation, load, and combinations thereof. The system will present 21

users with help dialogues and interactive tools to enable a simple step-by-step process, offering 22

customers tailored information based on their inputs to provide accurate information in the 23

interconnection request process. In this manner, GIPT will not only guide customers through a self-24

selected tariff application, it will also provide options as to which tariff would be best suited for their 25

needs. GIPT performs fast-track or detailed analysis during the customers’ interconnection request 26

submittal process, which greatly reduces the duration for the Permission to Operate (PTO) to be issued. 27

Based on immediate feedback, customers can make informed revisions to their project configuration 28

prior to submitting an interconnection request and payment, resulting in an increased interconnection 29

success rate. Digital payments combined with work management process and account receivables 30

process integration create a seamless financial pipeline so SCE’s processing work can begin more 31

148

quickly. Transparency in lifecycle project management and communication keeps all internal and 1

external stakeholders informed throughout the interconnection process. These efforts result in further 2

improvements to system forecasting, capacity planning, operations, public data publishing, Distribution 3

Resource Plan requirements, and customer satisfaction. 4

The GIPT, when combined with the other work streams listed above, will 5

provide more accurate interconnection responses in a shorter period and reduce the growing backlog of 6

interconnection requests. Once fully implemented, the GIPT has the potential to significantly decrease 7

the cycle time, or time between application submission and PTO for DER interconnection and 8

potentially reduce project costs. The GIPT will provide the following benefits: 9

• More accurate DER information will lead to a more accurate load 10

forecast, DER forecast, and DER adoption model. This will allow for 11

more accurate capacity planning. 12

• Improved load flow analysis, system impact studies, and integration 13

capacity analysis. A more complete representation of NEM generators 14

in the Grid Connectivity Model will improve results from the System 15

Modeling Tool.206 16

• Better DER locational data. System operators will have a clearer and 17

more comprehensive understanding of where DERs are on the grid. 18

Routine switching procedures will cause more predictable changes to 19

voltage and loading. An improved ability to accurately monitor and 20

enforce contractual operating parameters of DERs will support more 21

optimal grid conditions. 22

• Publishing up-to-date integration capacity analysis results will better 23

lead customers to areas of the grid better able to support their proposed 24

DER interconnection. This improved certainty of interconnection 25

viability can be obtained with a more comprehensive source of DER 26

data. 27

206 Please refer to SCE-02, Vol. 10 for a detailed description of the System Modeling Tool.

149

• By integrating with the System Modeling Tool, GIPT will decrease the 1

time required to perform a technical evaluation, generate reports, and 2

contracts. 3

• Reduced distribution upgrade costs. When customers leverage the 4

Integration Capacity Analysis results from SCE, a higher cost certainty 5

is achievable. The Integration Capacity Analysis describes the DERs 6

the system can handle in its existing configuration, without additional 7

distribution upgrades. Comprehensive DER data provided by GIPT 8

will enhance the accuracy of Integration Capacity Analysis. 9


SCE has conducted a pilot project to test the value of automating portions of the 11

interconnection process. This pilot has proven successful in digitizing and accelerating the application 12

intake process for behind-the-meter NEM interconnections. Through the pilot, SCE has also assessed its 13

current and desired interconnection lifecycle processes. 14

The GIPT Project Plan will prioritize functionality roll-out in the following 15

manner: 16

• 2016: Prepare multiple databases and validate the information. 17

• 2017: Deploy base product, configure for NEM and Rule 21 non-export to 18

prepare to retire pilot solution. 19

• 2018: Integrate with System Modeling Tool for automated analysis; leverage 20

ICA results to streamline interconnection process; configure tool to support 21

load interconnection, WDAT, and TOT to include cluster study analysis. 22

SCE forecasts $13.68 million to complete this scope of work. The capital forecast 23

for this project includes project team costs for SCE employees, supplemental workers, and consultants, 24

software and vendor costs, and hardware costs. The project estimation is based on our internal SCE IT 25

cost estimation process. See the workpaper for more detail.207 26


150


Alternative 1: SCE considered procuring a COTS solution for standalone 2

project management solutions and standalone workflow tools. We did not pursue this option as none of 3

the COTS products appear capable of dealing with the level of complexity, reducing the manual steps, or 4

providing the accurate information needed by SCE and customers. Most COTS products (unless 5

modified) would increase the number of manual work process steps due to the way they manage 6

workflow processes. These standalone tools are not expected to solve data integration challenges related 7

to data validation and accuracy issues because some of the third-party installers may need to request 8

additional information such as service account, address clarification, or verification of application 9

information (e.g., location, valid DER information) from customers. 10

Alternative 2: SCE also considered expanding the functionality of the 11

existing pilot application to support the full GIPT scope. We did not pursue this option as there are 12

multiple limitations of the pilot program, both in terms of functionality and in terms of technical 13

viability. After considering the dependent requirements in various other initiatives such as the System 14

Modeling Tool and the Long Term Planning tool, SCE determined this option is not viable. 15

21. Grid Analytics Applications (GAA) 16

Table V-45 Grid Analytics Applications208



The Grid Analytics Applications (GAA) will provide SCE engineers, operators, 18

and distribution grid designers with improved analytical, visualization, and decision-support capabilities 19

required to plan and operate the grid of the future. Investment in GAA is necessary to evaluate the 20

effects of higher DER penetration, accurately predict future conditions, and develop plans to better 21


Recorded Forecast

CIT-00-SD-PM-000247 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 3.74 12.73 0.36 0.37 0.39 17.59 Previous GRC Request - - - - - - - -

151

accommodate DERs into system operations. GAA will provide SCE personnel with the capability to 1

perform analytics on large data sources, including smart meter data, weather data, outage data, and 2

electrical network field measurement data (e.g., Supervisory Control and Data Acquisition, or SCADA, 3

data). In addition, the GAA will allow engineers to further analyze field measurements of the electrical 4

network for abnormal grid conditions and augment it with Smart Meter data to accurately predict asset 5

load profiles, which can then be used for long-term system-planning analysis. These applications also 6

allow SCE personnel to visualize the analytics on geographic information system maps and asses circuit 7

voltage degradation, line transformer utilization, and accuracy of transformer to meter relationships. 8

SCE personnel will also be able to perform more granular post-mortem analyses of outages to improve 9

current and future reliability management. For the sake of clarity, GAA will provide capabilities that are 10

separate and distinct from the CSAT project described above. Moreover, GAA will provide unique 11

analytics capabilities associated with planning and operating the distribution system, whereas the CSAT 12

project will provide unique analytics capabilities associated with the bulk power transmission system. 13


Implementation of Smart Meters has provided a significant opportunity to 15

leverage the historical hourly customer meter information (e.g., hourly peak demand, average voltage 16

reads, and energy demand usage) provided by these devices to better understand customer energy usage 17

patterns and overall status of the grid. This smart meter data, when integrated with electrical network 18

connectivity information and status of the grid assets, provides the most accurate information for 19

operators and engineers to make proactive and reactive decisions. This integrated data, made available 20

through GAA, will better inform SCE personnel on troubleshooting power outages and restorations and 21

power quality issues. Analytics can also be performed to better understand the asset load conditions to 22

proactively predict loss of life or catastrophic failure of assets before it occurs, enabling a proactive 23

corrective action. 24

To perform these types of analytics, key applications will be procured to enable 25

further detailed analysis of the electrical grid. A voltage analytics tool will be developed to provide a 26

more granular view into voltage characteristics across the distribution system, which will enable 27

enhanced system planning and operation of the grid. Load profile analytics using raw field measurement 28

data would be provided to system planners to develop foundational circuit load profiles. These profiles 29

can then develop system planning forecasts to analyze and compare DER-based solutions to traditional 30

wire-based solutions. In addition, load profile analytics can be leveraged to develop more realistic DER 31

152

device profiles. System planning requires comprehensive Photovoltaic (PV) profiles for determining 1

output capacity based on attributes such as local geographical solar insolation, local historical cloud 2

cover, and historical PV performance to supplement distribution system forecasts. DER profiles (from 3

GIPT data), and load profiles acquired from field measurement data, enable the ability to operate and 4

plan a dynamic distribution system capable of integrating large numbers of DERs and their impacts to 5

system planning performing a comprehensive capacity analysis. 6

The GAA enables the ability to develop analytics around assets to understand 7

characteristics such as historical load and DER adoption rates and operational asset overloads. Localized 8

load growth and DER adoption rates (using information from systems such as GIPT) can be integrated 9

to inform system planning forecasts to better understand future distribution grid needs by analyzing 10

historical installations throughout the SCE territory. In addition, during storm conditions (i.e., peak 11

conditions) grid operations could use the GAA in evaluating historical line transformer overloads, circuit 12

overloads, substation overloads, and at any aggregate level within the distribution grid using smart meter 13

data, and field measurement data to prioritize and communicate those overloads that may cause large 14

outages. 15

Today, SCE can perform grid analysis using distribution circuit and substation 16

data points to evaluate utility assets for overloading and voltage concerns for substations and circuit 17

mainlines. However, this analysis is conducted without the benefit of leveraging load and voltage 18

profiles available from customer smart meters. These raw data reads are used in system planning to 19

perform peak load forecasting for circuit and substation assets. In addition, availability of circuit and 20

substation level data points only provides a high-level overview of distribution grid peak demand, and 21

does not provide insight to individual customer’s peak demands and localized issues such as low 22

voltage. Manual labor-intensive processes are required to include customer meter information in the 23

analysis. Customer meter information only allows for a “point in time” view of customer behavior and 24

may not completely reflect the history required to both inform and proactively assess customer electrical 25

service and power quality issues. 26

(1) Benefits 27

Once the Grid Analytics Applications are functional, the following 28

benefits can be realized: 29

1) Analytics on customer meter data can be automated to perform routine 30

calculations such as voltage degradation (i.e., low/high voltage 31

153

evaluation), accurate line transformer loading, and historical outage 1

conditions. These capabilities proactively enable system planners and 2

operators to quickly evaluate portions of the grid that may be at risk of 3

exceeding acceptable operating thresholds and conditions. A system 4

planner can accurately assess the current loading of a line transformer 5

serving multiple customers (based on an automated aggregate service 6

customer meter data) and can adequately predict the effect of load and 7

generation increases on these transformers based on a new customer 8

request of installing a large motor, air conditioning unit, or DER. In 9

addition, grid operators during outages can assess the electrical service 10

status for groups of individual customer meters by evaluating meter 11

statuses through interactive GIS maps. 12

2) An analytics application using customer meter data can supplement 13

and automate missing historical circuit and substation operating data 14

points. This is required during abnormal circuit re-configurations and 15

when SCE initiated demand response events occur. The ability to 16

aggregate and blend collections of customer meter data with circuit 17

and substation load trends allows for more accurate system planning 18

during peak conditions. These newly blended historical load profiles 19

for SCE’s 4600+ circuits can then be leveraged by other tools such as 20

a Long Term Planning Tool (LTPT) to perform load and generation 21

forecasts to then evaluate mitigation for potential infrastructure 22

overloads. 23

3) The ability to develop time series profiles, such as load and DER 24

profiles, enables system planning and grid operations the insight to 25

further integrate DER installations with a future dynamic distribution 26

system. These profiles will accurately communicate how DER and 27

load demand may coincide to benefit or create issues on the 28

distribution system that will require the grid solutions to mitigate (e.g., 29

traditional capital projects). 30

154


The GAA will deliver the functional capabilities and will have the following 2

implementation plan: 3

• 2016-2019: Foundational Asset Analytics – Automated customer meter and 4

field measurement asset voltage and current analytics to inform SCE 5

distribution engineers, distribution grid designers, and grid operators of asset 6

violations. 7

• 2017-2020: Engineering Analytics – Ability to perform statistical analytics on 8

power system data (i.e., voltage deterioration and circuit load profiles) both 9

manually and as an automatic service. 10

• 2017-2018: Operational Analytics – Near real-time streaming analysis of grid 11

asset overloads and customer outage playback behavior to further enhance 12

operating the distribution grid through proactive mitigation. 13






Alternative 1: SCE considered not pursuing this project and continuing to 19

use existing manual data analysis methods for planning and operating the grid. We did not pursue this 20

option due to the vast amounts of electrical and asset data types, sources, and formats that SCE 21

maintains that require significant time and resources to access, consolidate, validate, and then analyze. 22

This approach does not allow SCE personnel to fully utilize SCE’s smart meter data to support grid 23

analytics at the customer and distribution circuit level. 24


155

22. Long-Term Planning Tools 1

Table V-46 Long-Term Planning Tools210



The Long-Term Planning Tools (LTPT) is a set of software tools that will 3

facilitate integrated planning and forecasting over a five-to-ten-year horizon to identify optimal solutions 4

to system planning challenges. These challenges include poor reliability and loading conditions created 5

when installed assets are expected to exceed design limits because of forecasted changes in loading and 6

generation. Functions of the LTPT include advanced circuit and substation modelling to support 7

distributed energy resources (DER) integration, power flow and system planning analyses, calculation of 8

load blocks at circuit and substation levels, and capacity planning analyses. LTPT is not a single stand-9

alone tool; rather, it is a compilation of several software products integrated together into a single 10

platform. LTPT will replace the existing suite of outdated software tools incapable of performing the 11

analyses needed for the modern electric grid. 12


Due to the limited capabilities of the current analysis and planning tool set, there 14

is the potential risk of introducing errors into SCE’s long-term plans resulting in over- or under-15

estimation of load and generation capacity. SCE must forecast the load and DER growth accurately and 16

quantify and rank all elements of project risk. Therefore, SCE must develop advanced system planning 17

methodologies, which require the implementation of new tools. 18

The need to replace the current suite of tools with a more complete solution 19

includes: technical obsolescence of the current tool, inability of current tools to integrate with other SCE 20

systems, and changes in the business needs and legislation that cannot be met with the existing tools. 21


156

Obsolescence 1

The current tools are approaching the end of their lives and are degrading rapidly 2

in terms of usefulness and maintainability. Current issues include: 3

• Difficulty to maintain. The current tools were built in-house during the last 4

decade using a complex architecture of custom-built code and business logic. 5

Over time, it is becoming increasingly difficult to maintain the skill sets 6

required to support the ongoing operation of this aging technology. 7

• Increased user base. The current tools were designed and built specifically for 8

the distribution circuit and substation projects. However, as SCE realized 9

operational efficiencies in integrating the work groups (e.g., system planning, 10

district (load growth) planning, programmatic projects, project management), 11

the number of users has increased by over 50 times. The underlying 12

architecture of the tool cannot handle this many users, which is impacting 13

system performance. 14

• Inability to scale. Besides the dramatic increase in users, the current system’s 15

database has grown to almost five times its original design limit. This is a 16

major architecture problem as the current tool maintains all projects (and their 17

alternatives) indefinitely. As the user base and number of projects expanded, 18

the database grew exponentially. 19

Inability to integrate with other systems 20

As SCE moves to an integrated approach to grid planning, our systems and 21

software need to interoperate with each other seamlessly. Examples include a unified analysis of 22

transmission, sub-transmission, and distribution needs from a “bottoms-up” and “top-down” analysis. 23

Functions such as risk-based analysis, protection engineering, and reliability studies need to be 24

integrated with the other analyses. The current tool lacks the ability to integrate with other important 25

systems (e.g., CYME211, PSLF212) and cannot be made to integrate without completely rebuilding much 26

of the architecture or using extensive manual processes. 27

211 The CYME software package, acquired in 2008, is used to calculate power flow and short-circuit duty for actual or simulated configurations and generation/loading conditions on distribution circuits.

157

• Sub-transmission and transmission analyses and project simulations are 1

performed independently using a separate set of software tools. In the LTPT 2

all will be performed in one integrated platform where manual re-entry is 3

eliminated and personnel can perform multiple iterations for optimizing their 4

solutions. 5

• The current tool does not have the capability of fully evaluating and 6

quantifying risks associated to each condition or proposed project. Since it 7

cannot integrate with the risk analysis tools, this function must be done 8

outside of the tool and the results analyzed independently. This lack of 9

integration creates inefficiencies as proposed projects must have their risk 10

assessments performed outside the tool and subsequently have the results 11

brought back into the tool. This problem is magnified by the iterative 12

approach required in analyzing risk for a large portfolio of projects. 13

• The current tool cannot integrate with other project management and financial 14

systems. 15

Changes in business needs. The increasing complexity of the electric grid 16

demands new cutting-edge methodologies to identify capital projects. The current tool lacks several 17

major required functions, including: 18

• The growth of DERs on our system will require us to evaluate the grid based 19

on load profiles as opposed to a single peak load value as done today. The 20

current tool is incapable of providing this functionality. 21

• There are multiple load forecasts used for different purposes at SCE. 22

Historically, these forecasts include a distribution forecast, and transmission 23

forecast, and a system-level forecast. These forecast are primarily developed 24

independently by different organizations and are not aligned. SCE must align 25

these forecasts to understand the implications that DERs may have on the 26

Continued from the previous page 212 Positive Sequence Load Flow (PSLF) is a software package that is used to calculate power flows on transmission and subtransmission systems.

158

distribution and bulk electric systems. To coordinate planning and 1

identification of optimal grid solutions and to align with the vision in the 2

Distribution Resources Plan (DRP), SCE must utilize a load and DER forecast 3

coordinated with the CEC Integrated Energy Policy Report (IEPR) and across 4

all internal planning organizations. 5

• The current planning tool performs system analysis assuming one-way power 6

flow. In this manner it can forecast potential problems such as circuit and 7

substation overloads based on loading limits and forecasted demand. The 8

current tool cannot perform analyses resulting from bi-directional flow213 and 9

cannot evaluate overloads at a more granular level such as a line segment. 10

• The current tool cannot evaluate the risk types associated with identifying 11

optimal grid projects/solutions. To enable a process for managing the elements 12

of the risks identified, LTPT will facilitate the concurrent evaluation of 13

multiple types of projects for risk mitigation, including utilizing DERs to defer 14

or replace traditional capital upgrades. System Planners can evaluate the 15

project driver and evaluate multiple project alternatives. System Planners can 16

better determine the value that DER can add to the distribution and 17

transmission systems by creating load profiles for DERs as well as customer 18

demand and evaluating the profiles over an annual period. 19

In conclusion, a profile-based, coordinated load and DER forecast will enable 20

SCE to determine where affected areas will be. A risk-based, coordinated planning process will enable 21

SCE to determine risks on the distribution grid. A methodology to develop optimal grid solutions to 22

address the identified risks will enable SCE to fully utilize DERs while developing a safe, reliable and 23

affordable future distribution grid. 24

213 Bi-directional power flow: Historically, power has flowed from the source (transmission system) to the load

(customers). In the present system, the high penetration of distributed generation can cause power to flow in the “reverse” direction, from the customer to upstream portions of the system. The existence of power flow in both directions is referred to as “two-way” or “bi-directional” power flow.

159

(1) Benefits 1

The LTPT will provide better response to customer requests, with the 2

GIPT discussed previously in this volume, allowing customers to get faster, more accurate answers, 3

reducing power quality issues, and increasing the ability to use the full hosting capacity of the 4

distribution system. The LTPT will also integrate with the Work Management’s Portfolio Management 5

project, described previously, for effective bundling of work to be performed in the field. This will allow 6

SCE to better utilize its field resources while maintaining reliable grid system. 7

i. Forecasting 8

The LTPT will increase the accuracy of SCE’s Load and DER 9

Forecast by incorporating load profiles versus a single point representation of load and generation. As 10

described above, this may cause better identifying mitigations, and may cause a reduction of the capital 11

upgrades identified using a point based forecast. 12

ii. System Analysis 13

The LTPT will expand SCE’s capability to perform analysis to 14

allow SCE engineers to proactively address forecasted grid concerns. This may cause reduction in 15

customer minutes of interruption (CMI). 16

The LTPT will allow planners to address multiple risks within a 17

project. This may prevent multiple projects in the same geographic area, which may cause fewer 18

customer outages, less cost due to coordinating resources, less cost due to avoiding duplicative scope. 19

The LTPT will automatically determine circuit reconfigurations 20

resulting in a time savings to distribution planners. 21

iii. Optimal Grid Solutions 22

The LTPT will allow planners to create optimal grid solutions. For 23

example, the LTPT may recommend the full utilization of available capacity, or the utilization of DER 24

to defer grid upgrades. 25


The Long Term Planning tool(s) will have the following implementation plan, 27

which includes the following functional capabilities to be provided to the Electric System Planning, 28

Long-Term Demand Forecasting, Strategy, Integrated Planning and Performance, and Grid Operations 29

departments: 30

160

• 2016-2017: Load and DER profile-based Forecasting, Aggregation, 1

Distribution System Analysis, Capacity Analysis, Protection Analysis, Basic 2

Distribution Reliability Analysis, Optimal Grid Solutions, Basic Risk Model, 3

Basic Geographical Modeling, and Basic Integration with internal tools, Basic 4

Real Time Analysis. 5

• 2018-2019: Sub Transmission Analysis, Transmission Analysis, Advanced 6

Distribution System Analysis, Advanced Risk Model, Advance Geographical 7

Modeling, Full Integration with internal tools, Full Integration with external 8

tools. 9

The LTPT must be fully integrated with the planning tools and operational tools 10

described in this GRC, including the Grid Analytics Application, Grid Connectivity Model, System 11

Modeling Tool,214 Grid Interconnection Processing Tool, and the Grid Management System.215 The tool 12

must be integrated with the Grid Connectivity Model, the Grid Interconnection Planning Tool, and the 13

System Modeling Tool to minimize the amount of rework that must be done for each project and reduce 14

the amount of field data collection. The LTPT supports adding extensions into the Grid Connectivity 15

Model, again reducing the manual rework required and helping to keep the connectivity model accurate. 16

The LTPT solution will allow SCE to develop a load, DER and generation 17

forecast, perform system analysis to identify all grid risks, and determine optimal solutions to reliability 18

risks for the entire power grid from the lower end of distribution to the Bulk Transmission system. This 19

allows SCE to fully evaluate risks and opportunities related to existing and forecasted DERs in its long-20

term system planning process. The Long Term Planning Tool will allow SCE to develop and continue to 21

shape a safer, more reliable and affordable grid while encouraging DERs and enabling customer choice. 22





214 Please refer to SCE-02, Vol. 10 for a detailed description of the System Modeling Tool. 215 Please refer to SCE-02, Vol. 10 for a detailed description of the Grid Management System. 216 Refer to WP SCE-04, Vol. 2 Bk C p. 139.

161


Alternative 1: SCE considered pursuing the enhancement of an existing 2

distribution system planning COTS solution to meet LTPT functional requirements. We did not pursue 3

this option because no product in the marketplace possesses the full suite of capabilities required by 4

LTPT (i.e., capacity planning, project alternatives analysis for DER versus wires-based solutions, risk 5

analysis). Due to the level of complexity required to fully implement the LTPT solution, a complete re-6

design of both the technology and planning methodologies would be necessary to fully address 7

enterprise scalability and utility system planning requirements. Based on the large amount of 8

customization expected, SCE does not recommend this solution approach. 9

Alternative 2: SCE considered modifying the existing planning tool 10

currently used by SCE and enhance it to meet the new requirements. We did not pursue this option 11

because the current software solution cannot meet the existing system planning requirements and has 12

numerous performance issues. This option would require SCE to locate and identify qualified resources 13

and onboard them, as the technology is seldom used in practice in today’s industry technology 14

landscape. Based on our research, the LTPT project as presented here is the most responsive option that 15

SCE has found. 16

23. Grid Connectivity Model 17

Table V-47 Grid Connectivity Model217



The Grid Connectivity Model (GCM) is a software model of SCE’s complete 19

electrical grid. This model replaces existing disparate and disconnected models and will serve as the 20

single, centralized source of connectivity information for all assets—from bulk generation down to the 21


Recorded Forecast

CIT-00-OP-NS-000521 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 3.44 5.02 5.05 1.43 - 14.95 Previous GRC Request - - - - - - - -

162

end-consumer meters. This new software model will support other enterprise tools that require the use of 1

SCE’s electric system connectivity information and the operational configuration of devices.218 To 2

provide up-to-date information about the grid, this connectivity model will receive near real-time 3

information about device settings219 through multiple operational systems, such as the Distribution 4

Management System (DMS). 5


SCE’s current grid connectivity information is stored across different software 7

solutions, including separate electrical-based and structural-based solutions. Business processes around 8

these datasets vary across the owners of data, and automated synchronization of datasets is limited. In 9

the current state of the network model, gaps exist in three broad areas: 10

• No existing systems of record, 11

• Lack of information and inconsistent data, and 12

• Business process gaps in maintaining and synchronizing information across 13

systems. 14

The GCM represents the network of electrical devices and structural components 15

connected by conductors, in terms of nodes and links. The model considers various device statuses and 16

the electrical characteristics to represent the power flow network.220 The model will maintain various 17

perspectives of the grid, such as the as-built configuration, which represents the grid in its permanent 18

static configuration, and the as-operated configuration, which accounts for operational deviations from 19

the as-built permanent configuration of the circuit due to abnormal conditions or switching. Other 20

perspectives include historical as-operated views, and the as-planned view, which depicts the long-term 21

configuration of the distribution systems. 22

Grid connectivity information is currently scattered across multiple systems such 23

that no unified end-to-end connectivity model exists. SCE’s mapping organization uses a legacy system 24

for maintaining the electrical connectivity model for the schematic view of the distribution circuits. The 25

218 The GCM will support many tools and systems, including: planning tools (like CYME), analytical systems

(like Hadoop), and operational systems, such as the Outage Management System (OMS). 219 Examples of these device settings include capacitor bank settings and auto-recloser settings. 220 A distribution circuit model is composed of the following: connectivity model, asset data (nameplates,

ratings), device settings (capacitor bank operating limits, circuit breaker trip settings, automatic recloser trip settings), and planning and operation data (such as forecast electrical models).

163

transmission group uses the Energy Management System (EMS) system for static and dynamic views of 1

the transmission connectivity model. Grid operators use the distribution connectivity model in the 2

Outage Management System (OMS) system for managing outages and determining the extent of an 3

outage. Customer Service uses pieces of distribution secondary circuit connectivity information in 4

Customer Service System (CSS) and information from OMS for identifying the list of customers 5

impacted by an outage. The above business functions and roles will benefit from having a centralized 6

and unified grid connectivity model. 7

Engineering, design, construction, operations, maintenance, and customer service 8

departments rely on various network models manually created by various users for different analysis 9

required to operate the electric system. It is becoming more resource-intensive for users of network 10

models to manually gather data from different sources and analyze the distribution system. Today, the 11

attributes of various distribution assets and the grid require significant manual validation to verify that 12

the electric system models represent actual field conditions. There is no automated and reliable way of 13

inputting the actual device settings from the field into the connectivity model. These device settings are 14

important in simulations because the results should represent what is achievable in the electric system. 15

Verifying this information for one distribution circuit takes an engineer roughly five hours to complete. 16

It would be labor intensive and expensive for SCE to manually verify and model each of its distribution 17

circuits for the entire territory. The GCM will be integrated with operational systems like the 18

Distribution Management System (DMS) so near real-time data from DMS will provide actual device 19

settings information to the Grid Connectivity Model. 20

(1) Benefits 21

This project will provide several benefits by addressing the gaps in the 22

current state of the network model. It will eliminate the need to manually update various connectivity 23

models by interfacing directly with operational systems for device setting information. This results in 24

consistent connectivity data across systems and user groups. The GCM will also contain information 25

about distributed energy resources (DERs), collected from the customer-facing Grid Interconnection 26

Processing Tool (GIPT). Both grid-connected and behind-the-meter DERs will be modeled with their 27

interconnection information (e.g., generating capacity, technology, contractual parameters) in the overall 28

connectivity model. 29

The GCM will provide the system of record for reliable, up-to-date, and 30

end-to-end connectivity model of the entire SCE grid such that various stakeholders (including system 31

164

planners, distribution engineers, and system operators) will have consistent information. The model will 1

be the system of record for all the connectivity information and allow users to effectively and reliably 2

manage the grid across planning, analysis, and operations. The project will leverage the existing GIS 3

system to provide an end-to-end view of the grid: from transmission to the distribution secondary. 4

Two models will be included as part of the GCM: the electrical 5

connectivity model and a structural connectivity model. The electrical connectivity model will provide 6

various business capabilities including tracing of the circuits, simulation capabilities, topology of the 7

entire grid, end-to-end connectivity information, schematic diagram generation capability for circuits, 8

geographic view of the grid, etc. The structural connectivity model represents various structures located 9

overhead (e.g., poles, towers, vaults) and underground (e.g., ducts and trenches). The structural 10

connectivity model will include information like the length of the conductor, impedance calculation of 11

segments, capacity of the underground ducts, location of electrical assets for maintenance, etc. The 12

structural connectivity model will complement the electrical connectivity model and enable various 13

electrical planning and analysis capabilities. 14

The Grid Connectivity Model will provide various perspectives of the 15

grid: as-planned, as-designed, as-built and as-operated. The as-planned perspective will provide the 16

long-term (5 to 10 years) view of potential future system configurations. The as-designed perspective 17

will provide the approved construction designs of the grid. The as-built perspective will be the base 18

foundational model that will provide a view of the grid, as it is constructed in the field. The as-operated 19

perspective will provide the current configuration of the grid. 20

The GCM will be the system of record for providing complete, correct, 21

and current information of the electrical and structural network connectivity, therefore enabling users to 22

perform more efficient, intelligent system operations. The Grid Connectivity Model can represent the 23

network connectivity in multiple perspectives, such as the as-built perspective and the as-operated 24

perspective. By looking at multiple system perspectives, grid operators can decide that are more 25

efficient. 26

Grid Connectivity Model will provide the connectivity information to the 27

rest of the enterprise using industry standards like Common Information Model (CIM). This will 28

promote the reuse of interfaces for GCM across the enterprise and eliminate the need for developing 29

custom interfaces for each new system. Web-based user interfaces will be provided to enable users to 30

165

view the schematic diagrams, perform tracing functions, perform simulations, and view the geographic 1

location of circuits. This information will be easily accessible to multiple user groups. 2


Phase I Scope 2016: 4

Phase I of the project will focus on addressing the needs of the Integration 5

Capacity Analysis (ICA), where the Grid Connectivity Model will provide connectivity information and 6

load information for performing ICA on distribution circuits as mandated by the CPUC. This will 7

include providing the as-built connectivity information and field device setting information for capacitor 8

banks and automatic re-closers from DMS. Load information will include monthly load profiles from 9

AMI data, DER generation data and circuit SCADA data for demand information. 10

Phase II Scope 2017: 11

Phase II of the project will focus on developing the substations’ internal 12

connectivity model and the foundational framework for the end-state design of the solution. We will 13

develop basic functionality for tracing and initial version of core engine for the Grid Connectivity 14

Model. This phase will also provide CIM-based connectivity files for the usage across SCE. For the sake 15

of clarity, the GCM will leverage the foundational work on the Trans-Sub Connectivity model 16

performed under the OMS Refresh project discussed above. 17

The scope will also include supporting various Grid Connectivity-related use 18

cases for outage management enhancement work, System Modeling Tool and ICA phase 2, Long Term 19

Planning Tool and the GIPT. 20

Phase III Scope 2018: 21

Phase III will include modeling the mesh networks and micro grids. It will 22

address the gaps in the secondary circuit connectivity information for transformer-to-meter connectivity. 23

A validation engine will be developed for verifying the connectivity model 24

changes. Phase III will also provide standards-based data services for the connectivity information to be 25

used by SCE. It will also provide schematic generation capabilities to address the requirements for as-26

operated, as-planned and as-designed perspectives. 27



166





needs of the GCM project. We did not pursue this option because there is no commercially-available 5

product available that can address the needs of Grid Connectivity Model comprehensively. The market 6

is not mature enough to provide a commercial product that can cover multiple perspectives of the 7

network. Various commercial products are specialized for a specific segment of the grid with a purpose. 8

This approach does not meet the requirement of a unified model where different users can benefit from 9

the information beyond their organizational or functional boundaries. 10

Alternative 2: SCE considered not pursuing this project and continuing to 11

use existing grid connectivity information stored across different software solutions. We did not pursue 12

this option because it will not address the various business capabilities that SCE planners and grid 13

operators need in response to the more complex grid and increasing number of DERs on the grid. This 14

approach will require significant time and resources to manually collect, consolidate, and validate the 15

combination of electrical and structural connectivity information before it can be used to support grid 16

planning and operations. 17


167

24. Transmission and Distribution Projects less than $3 Million 1

Table V-48 Transmission & Distribution Projects less than $3M222

Work Breakdown Structure (WBS) Forecast Capital Expenditures (Nominal $ Millions)

Table V-48 above lists capitalized software projects with total project costs less than 2

three million dollars in capital funding and will start and finish within the years 2016 to 2020. Please see 3

the workpapers for these projects for detailed information on each project and its associated costs. 4


WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-SD-PM-000187 Substation Health Assessment Tool (previously As - - 2.60 - - 2.60 CIT-00-SD-PM-000206 Substation 3D Design - - 1.26 0.72 - 1.98 CIT-00-SD-PM-000233 Electronic WO Package Automation (e-WOP) - - - - 1.00 1.00 CIT-00-SD-PM-000234 Fast Response Energy Storage 0.63 - - - - 0.63 CIT-00-DM-DM-000076 Transient Devices (HW for TSFT) 1.20 0.33 - - - 1.53 CIT-00-DM-DM-000083 High-Z Impedance Fault Detection - - - 1.00 1.00 2.00 CIT-00-DM-DM-000084 Secure DNP Ver5 Support for EMS - - - 1.00 - 1.00 CIT-00-DM-DM-000086 Grid Management Dashboards - 2.00 - - - 2.00 CIT-00-DM-DM-000090 PSMP 2.0 - - 1.00 1.00 - 2.00

Total 1.83 2.33 4.86 3.72 2.00 14.74

168

C. Power Supply Software Projects 223 1

1. Generation Automation Upgrade & Control Systems Refresh 2

Table V-49 Generation Automation Upgrade & Control Systems Refresh224



SCE owns and operates 33 hydroelectric (Hydro) generating power plants, which 4

can be divided into two groupings, Big Creek (formally known as Northern) and all Other (formally 5

known as Eastern) assets. Big Creek assets are the larger group, encompassing all SCE Hydro facilities 6

in the upper San Joaquin River watershed. These assets are in the western Sierra Nevada Mountains, 7

across an area centered approximately 50 miles northeast of Fresno. SCE’s Other Hydro assets comprise 8

the remaining facilities and are in the Bishop and Mono Basin areas of the eastern Sierra Nevada 9

Mountains, the Kern River, Kaweah, and Tule River areas in the southern Sierra Nevada Mountains, and 10

the Ontario, San Bernardino, and Banning areas in the San Bernardino and San Gabriel Mountains, 11

located in Southern California. Each power plant is equipped with various control systems to perform 12

and supervise the safe operation of the plant, and capture various data about the operation of the plant. 13

The control systems at the Hydro facilities have historically been sourced from 14

multiple vendors, and the lack of equipment standardization has caused problems with inter-operability 15

between the different Hydro operating regions and maintainability as components approach the end of 16

their life-cycle. SCE’s plans to standardize future plant control system equipment and software from a 17

single vendor. We have begun this process already. Standardization will eliminate the need to store parts 18

from multiple vendors and will improve communications between equipment. 19


CIT-00-SD-PM-000227 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 1.00 3.00 2.00 1.50 2.50 10.00 Previous GRC Request - - - - - - - -

Recorded Forecast

169


Replacing aged equipment must keep pace with current and future demands of the 2

system. Following replacement, the new automation system will provide vital information required for 3

operations, work management planning, preventative maintenance, regulatory compliance, and other 4

needs that cannot be conveyed with the existing communication system. This data can be captured, 5

transmitted, and retained with the planned automation upgrades. 6

(1) Benefits 7

Performing this Automation upgrade will add additional business 8

capabilities to SCE’s Hydro facilities, which include: 9

1. Enhancing the responsiveness of the Big Creek plant to CAISO control 10

signals by moving existing custom-developed code into the control systems. This will enable Big Creek 11

to offer more capacity into the CAISO regulation market, which will benefit the integration of renewable 12

generation into SCE’s grid; 13

2. Upgrading the control systems at Big Creek to avoid generator rough 14

loading zones (i.e., output ranges where the mechanical construction of the generating unit causes 15

excessive vibration); and 16

3. Providing a common test environment and enabling the interoperability 17

of the control systems in the different hydro operating regions to facilitate data exchange and 18

preparedness for enhanced disaster recovery capabilities between the regions. 19

Upgrades to the existing Automation system will enable Hydro to meet 20

mandates from FERC and other governmental entities to monitor water release capabilities and also 21

maintain dam surveillance. The overall benefit will be increased water control, system reliability, and 22

regulatory compliance. The Dam sites are important because they are points of mandated FERC 23

Instream Releases critical to FERC license compliance. 24

c) Project Scope and Forecast 25

This project continues the Automation standardization efforts, which include 26

implementing enhanced business capabilities and replacing existing automation systems as they become 27

obsolete. The Automation project will deploy advanced data archiving, monitoring, reporting, and 28

remote diagnostics with secure, NERC Critical Infrastructure Protection (CIP) compliant software and 29

servers for viewing of Ovation screens, data trending, and reporting on the administration network. This 30

project is driven by three business needs: (1) standardizing equipment, (2) replacing aged equipment in 31

170

use, and (3) adding new equipment to those areas not automated, which will allow for remote 1

monitoring and control of Hydro facilities. 2

The forecast project costs are $10.0 million.225 The capital forecast for this project 3







chose not to pursue this project, then we would have to keep the existing equipment in-service. This is 10

not a long-term viable option as software upgrades for the existing equipment are no longer being 11

supported by the vendor and will soon cease to meet current NERC CIP requirements. 12


upgrading the equipment using the existing vendor to meet the needs of the generation business. We did 14

not pursue this option because it would not meet our needs of providing standardization across SCE’s 15

generation assets while meeting NERC CIP requirements. 16

2. Dam Monitoring and Surveillance 17

Table V-50 Dam Monitoring and Surveillance226



SCE Hydro operates and maintains 22 dams considered by FERC to be large 19

and/or high-hazard dams, and many are in mountainous terrain at elevations over 7,000 feet above sea 20


CIT-00-SD-PM-000228 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 1.00 2.00 1.50 1.00 5.50 Previous GRC Request - - - - - - - -

Recorded Forecast

171

level. These locations are often remote and difficult places to work. SCE’s dams are under FERC 1

jurisdiction, and FERC has an ongoing effort to increase the safety of dams across the U.S. by 2

improving the monitoring of the dams and by improving the dams’ ability to withstand natural disasters, 3

including seismic events (earthquakes) and severe storms. 4

Several of our dams are both high-risk and time-sensitive. Time-sensitive means 5

there is not enough time to notify emergency services in case of a dam failure with sufficient lead time 6

to evacuate downstream towns. SCE monitors its dams through remote sensors that measure water levels 7

and water flows. When a sensor returns an unexpected or abnormal reading (which frequently occurs), 8

an operator must travel to the dam to perform a visual inspect. Often the travel time required can exceed 9

two hours, which in an emergency condition is not adequate. 10

This project captures the IT components of the Dam Monitoring and Surveillance 11

(e.g., cameras and communications equipment) Project. Supporting testimony for SCE’s risk analysis, 12

construction of field towers, permitting, and related building materials is captured within the SCE-05, 13

Volume 3 - Hydro O&M and Capital testimony. Two site surveys performed in 2015 assessed 14

equipment needs and were utilized to develop the initial project cost estimates. The total capital forecast 15

for the project described in the SCE-05, Volume 3 is $7.90 million, and the IT portion of this project is 16

$5.50 million for 2016-2020. 17

b) Project Need 18

SCE’s dams are under FERC jurisdiction, and FERC has an ongoing effort to 19

increase the safety of dams across the U.S. through improved monitoring and improvements to a dam’s 20

ability to withstand natural disasters, including seismic events (earthquakes) and severe storms. 21

(1) Benefits 22

The primary benefit for this project is improved response time to a 23

pending or developing dam failure by adding visual surveillance capabilities to high-risk dams, and to 24

remove the need to send operations personnel to perform visual inspections when abnormal sensor 25

readings are received. 26


The project scope at each dam will be similar and includes the purchase and 28

installation of cameras and communications infrastructure. Cameras require the ability to capture 29

periodic still images (e.g., 5 frames per minute) and remotely capture a still image on demand 30

(sometimes infrared vision will be required to capture images at night or in snowy conditions). Cameras 31

172

and communications equipment must remain operable in remote areas and under harsh weather 1

conditions. In addition, not all dams have electric power and necessary communications infrastructure so 2

this will be provided via solar and/or wind generation and satellite and/or microwave communications. 3

Installation will occur according to the risk priority assigned to each dam and some installations may 4

span multiple years (i.e., project may commence in the fall of the first year and conclude in the second). 5

The actual cost per dam will vary across different dams depending on dam size 6

(larger dams will require more cameras, more cabling and/or wireless communication links between the 7

cameras and satellite dish), communications availability (some dams have existing microwave 8

communications equipment with capacity, other dams have no communications infrastructure and 9

require a satellite link due to their remote location), power availability (some dams have electric power 10

at or near the dam, other dams have no power infrastructure and will need to be powered via solar and/or 11

wind power generators with batteries for energy storage). 12





workpaper for the cost breakdown information.227 17


Alternative 1: SCE considered installation of fixed power and fiber-optic 19

communications links to each dam. Initial cost estimates performed in 2012 showed that the fixed power 20

and fiber-optics communications link would not be a lower-cost option. Communications equipment 21

costs were $2.4 million for these seven dams; assuming similar complexity for the other dams, the total 22

project cost estimate was approximately $8.2 million. 23

Alternative 2: SCE considered the use of remote-control unmanned 24

aircraft (i.e., drones) fitted with cameras to monitor the dams. Use of drones was deemed to not be a 25

viable option due to it being an unproven technology for this application. Currently, FAA regulations 26

require drones to be flown only within line-of-sight, which limits their usefulness in remote areas. In 27

addition, a drone may not be able to fly (or survive the flight without crashing) in harsh weather 28


173

conditions, including heavy rain, cold, and snow. It is also unclear if commercially available drones 1

exist that could satisfy the business needs listed above (and public safety requirements). In addition, 2

some of the SCE dams are located on land managed by the U.S. Forest Service, and the regulatory 3

environment for drones on federally managed lands is evolving. For these reasons, this alternative was 4

not pursued. 5

3. CAISO Market Enhancements Program (IMEP) 6

Table V-51 CAISO Market Enhancement Program (IMEP) 228

Work Breakdown Structure (WBS) Authorized Recorded, & Forecast Capital Expenditures (Nominal $Millions)


D.15-11-021 adopted the CAISO Market Enhancement program as part of SCE’s 8

2015 GRC. For 2013 and 2014 this project was funded through the MRTU Memorandum Account and 9

not through the GRC. In 2015 the program was funded through the GRC. This program was previously 10

presented as two projects in SCE’s 2015 GRC application: (1) CAISO Market Enhancements Project 11

(2013); and, (2) CAISO Market Enhancements Project (2014 – 2017).229 Each CAISO release 12

implements several market enhancements. 13

CAISO conducts an annual process to identify and prioritize future market 14

initiatives. Once identified, each of the initiatives is ranked based on two criteria: (1) benefits, including 15

grid reliability, overall market efficiency, and desire by market participants; and (2) feasibility, including 16

market participants’ implementation impact (costs and resources) and CAISO implementation impact 17

(costs and resources). 18

228 Refer to WP SCE-04, Vol. 2 Bk C pp. 204-211. 229 Please see SCE’s 2015 GRC Application – SCE-05, Volume 2, Part 1, pp. 189-200.

Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast - 1.42 1.60 2.16 2.16 3.95 4.00 4.00 4.00 4.00 Previous GRC Request* - - 5.30 6.79 7.00 7.00 6.00 *The Commission adopted SCE's 2013 recorded costs and 2014-2015 request for this project in D.15-11-021.

Various WBS IDs include: CIT-00-SD-PM-000225, CIT-00-SD-PM-000162, CIT-00-SD-PM-000069, CIT-00-SD-PM-000164, CIT-00-SD-PM-000212

Recorded Forecast

174

The CAISO Stakeholder Initiatives Catalog230 outlines the priority issues 1

identified by the CAISO and stakeholders that may require enhancements to the energy market. This list 2

is used by SCE to identify market initiatives that have a high likelihood of being addressed in the 2016-3

2020 timeframe. Although the exact timelines and requirements are still being defined by CAISO, SCE 4

must be able to respond and implement required system enhancements once these market initiatives are 5

mandated by CAISO. 6

Based on the catalog, CAISO plans to make significant enhancements to their 7

Day Ahead and Real Time Markets during 2016-2020 timeframe. CAISO is also planning to enhance 8

some of their major processes including Residual Unit Commitment, Ancillary Services procurement, 9

Convergence (Virtual) Bidding, Congestion Revenue Rights procurement, and Resource Adequacy. 10

Other initiatives on the roadmap will enhance the Capacity Procurement 11

Mechanism, implement the Reliability Services initiative, replace the existing Operational Meter 12

Analysis and Reporting (OMAR) meter data management system, and enhance the way CAISO handles 13

Multi-Stage Generators. The CAISO multi-state Energy Imbalance Market (EIM) will be enhanced, and 14

new participants are expected to join the EIM market. 15

The objective of the CAISO Market Enhancements Program is to enhance 16

existing systems and processes to implement CAISO market initiatives to meet SCE’s CAISO 17

Scheduling Coordinator obligations. This implements ongoing CAISO market changes that happen 18

every year. 19


SCE is a CAISO participant and subject to the CAISO’s tariff and operating 21

procedures. Therefore, SCE must update existing systems to implement CAISO market changes to 22

remain compliant with CAISO tariffs and participate in the CAISO market. 23

(1) Benefits 24

This project will support compliance with the CAISO tariff provisions. 25


In 2015, SCE recorded lower expenditures than authorized primarily because 27

CAISO requirements changed between our GRC application and when the project was implemented. 28

230 Catalog available at

http://www.caiso.com/Documents/Final_2016StakeholderInitiativesCatalog_Roadmap.pdf.

175

These changes resulted in a lower-than-expected project complexity. The project spend varies year by 1

year depending on the size and complexity of the CAISO changes implemented. 2

Since implementing the CAISO Market Redesign and Technology Upgrade 3

(MRTU) project in 2009, the CAISO energy market continues to evolve. SCE will need to modify its 4

existing systems to handle the required changes according to the latest CAISO roadmap. 5

SCE forecasts the IMEP project to be $19.8 million for the 2016-2020 period.231 6



components. SCE’s forecast for this project includes costs for SCE employees, supplemental workers, 9

and consultants, software and vendor costs, and hardware costs. See this project’s workpaper for the cost 10

breakdown information. 11

Based on an analysis of the information from CAISO about market changes in 12

this period, SCE has estimated this project to be a high complexity COTS software implementation 13

requiring significant upgrades to existing applications, the system interfaces that connects the systems 14

and databases, and hardware infrastructure. 15


This project implements tariff changes CAISO, which are required for 17

regulatory compliance. Therefore, no alternatives have been considered. 18


176

4. Energy Planning Platform Upgrade (EPP) 1

Table V-52 Energy Planning Platform Upgrade232



This project will upgrade the existing Energy Planning Platform application to 3

incorporate new market requirements and reporting functionalities. SCE implemented this application in 4

2013 to support portfolio analysis and regulatory reporting and to provide advanced analytics for SCE’s 5

energy hedging program. The original application has grown from a single risk-reporting module to 6

several complex analytical modules. This project will provide the EPP application with the upgrades 7

required in 2018-2019. 8


As SCE’s power procurement portfolio continues to change, so does its reporting 10

requirements. As Community Choice Aggregators (CCAs) are established in the SCE service territory 11

there is a need to map and track the energy and customers served by these CCAs. SCE also expects the 12

need to track resources installed on the distribution grid, usually referred as distributed energy resources 13

(DERs). 14

SCE requires a refresh of its analytic platform to accurately capture SCE’s market 15

position (i.e., how much supply is available vs. the how much demand is projected at different times and 16

locations). Additionally, SCE must be able to track and report on CCAs and expected future distributed 17

energy resources, and to be able to report on how different forecasts for future market prices will impact 18

the cost of supplying power to customers. 19


CIT-00-DM-DM-000078 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 1.13 1.19 0.58 0.29 - - - 2.00 2.00 - *2011-2014 recorded costs pertain to previous phases of the EPP application, and do not pertain to this GRC's EPP Upgrade project.

Recorded Forecast

177

Furthermore, SCE is also seeking to retire some of its older applications and move 1

the functionality to the new EPP system. This will create a more robust data management process and 2

will also avoid future IT maintenance costs for the systems that are replaced by the new EPP system. 3

(1) Benefits 4

This project will enhance SCE’s energy portfolio reporting capabilities. 5


SCE forecasts total project costs of $4.0 million over the 2018-2019 period to 7

upgrade the EPP.233 The capital forecast for this project was developed using SCE’s internal cost 8







chose not to pursue this project, then we would not be able to accurately monitor SCE’s market position, 15

track and report on CCAs and expected future distributed energy resources, and report on how different 16

forecasts for future market prices will impact the cost of supplying power to customers. 17


178

5. PCI Replacement 1

Table V-53 PCI Replacement234



D.15-11-021 adopted the Power Costs, Inc. (PCI) Replacement project, where it 3

was referenced as the Market Systems Replacement project, with a forecast of $5.8 million. The project 4

start was delayed due to dependencies on the Commodity Management Platform (CMP) project that 5

implements a new system to manage renewable power contracts. The CMP project is scheduled to be 6

fully implemented in 2016. 7

The existing system is used to enable SCE to participate in the California 8

Independent System Operator (CAISO) energy market. The system was installed at SCE in 2009 and 9

manages generator outages, captures power trades, creates and submits energy bids to the CAISO energy 10

markets to purchase and sell electric power, and downloads market awards and market prices from 11

CAISO. 12

This project will replace the existing system with new third-party vendor 13

solutions selected through a competitive solicitation process. Since the installation of the existing 14

system, the software vendor landscape has changed significantly; several vendors have merged and 15

continued product development has been done by multiple vendors to enhance the integration, 16

performance, and analytics offered by their systems. 17

This project is related, yet distinctly different, to the CAISO Market Enhancement 18

(IMEP) project described above. While both projects relate to CAISO, the IMEP project implements 19

ongoing market changes mandated by CAISO, whereas this project is a one-time effort to replace the 20

system platform used for day-to-day operations. 21


CIT-00-DM-DM-000042 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.00 3.50 1.00 - 7.50

Previous GRC Request - - - - - 3.00 2.80 5.80

Recorded Forecast

179


There are issues with the current system that must be addressed by replacing this 2

system with a newer platform. These issues include a lack of ability to keep up with CAISO market 3

changes, system performance, and limited functionality for analytics. In addition, several vendors now 4

offer hosted solutions that provide the same or better functionality at a lower cost, compared to 5

traditional on-premises solutions. 6

SCE requires enhanced system functionality in the following areas: 7

• Ability to participate in CAISO market simulations. CAISO market changes 8

are implemented through a cycle consisting of stakeholder processes to design the market changes, 9

workshops to review the detailed technical design of the market changes, market simulation of the 10

changes, and finally go-live where the changes are implemented into the production environment. The 11

purpose of market simulation is for both CAISO and market participants (including SCE) to test the new 12

market changes using systems and business processes that closely, if not entirely, resemble those 13

actually used to perform the transactions in production. In the past, SCE has frequently had to perform 14

market simulations using manual workarounds instead of the actual software since our vendor could not 15

deliver software updates in time for the CAISO market simulations. 16

• Adequate performance in the SCE environment with data volumes as required 17

by CAISO. CAISO data volumes have increased dramatically after the Market Redesign and 18

Technology Upgrade (MRTU) market went live in 2009. SCE has seen issues where our current system 19

struggles to retrieve and process the CAISO data volumes in a timely manner. 20

• Better functionality for analytics. Analytics is increasingly important as the 21

CAISO market grows more complex and the portfolio of renewable power continues to grow. Adequate 22

analytics require access to updated and relevant data using a variety of data reporting and analysis tools. 23

SCE has experienced issues where data is stored in the current vendor database in a format either 24

inaccessible or hard to retrieve and analyze using industry-standard reporting and analysis tools. 25

(1) Benefits 26

This project will replace the existing system with a new system that will 27

offer similar or better functionality at lower total cost of ownership. 28


When the existing system was installed at SCE in 2009, the industry-standard 30

approach to implement new software was to run the software on servers in SCE’s data centers. Since 31

180

2009, significant technological development has taken place among software vendors and one of the 1

most important evolutions has been the introduction of hosted or cloud solutions. 2

Hosted solutions offer several advantages to SCE for this project: 3

• Lower initial installation costs since computer hardware does not have to be 4

purchased and the software does not have to be manually installed on the new 5

hardware. 6

• Avoided ongoing operating costs since the vendor will operate SCE’s system 7

in parallel with systems from other clients, which offers opportunities for 8

automating similar maintenance procedures across different client systems. 9

• Faster and more effective response to CAISO market changes. Today SCE 10

must receive new software releases from the vendor, test the software, and 11

implement it into our computer environment before the users can perform 12

their testing and/or participate in CAISO market simulations for new energy 13

market products. In a hosted solution the vendor tests and installs the new 14

software release on their servers, which reduces the overall cycle time. This 15

allows SCE more time to participate in market simulations. 16

The market survey and analysis that SCE conducted in 2014 showed that vendor 17

solutions are available that offer similar or better functionality at lower total cost of ownership than 18

today’s solution.235 19

In the 2018 GRC the total project costs are $7.5 million. The capital forecast for 20

this project was developed using SCE’s internal cost estimation model. This model utilizes industry best 21



costs, and hardware costs. See this project’s workpaper for the cost breakdown information.236 24

235 Refer to WP SCE-04, Vol. 2 Bk C pp. 225-236. 236 Refer to WP SCE-04, Vol. 2 Bk C p. 224.

181



chose not to pursue this project, then we would incur higher total five-year ownership costs based on 3

vendor assessment and analysis performed by SCE in 2014 and 2015. 4

Alternative 2: SCE considered custom development to meet the business 5

needs. We did not pursue this option since the third-party vendor offerings are mature and will meet our 6

business needs. If we developed a custom solution we would need to perform ongoing development to 7

implement planned CAISO changes and would not benefit from sharing these costs with multiple other 8

customers, as we would be able to do with a third-party vendor. 9

6. Energy Trading and Risk Management (ETRM) System Replacement 10

Table V-54 Energy Trading and Risk Management237



D.15-11-021 adopted the Energy Trading and Risk Management (ETRM) System 12

Replacement project as part of SCE’s 2015 GRC. The ETRM System Replacement project will replace 13

the existing ETRM system installed in 2008. It is used to store and process all conventional bilateral 14

power contracts and physical and financial power and gas transactions. 15

The project start was delayed due to dependencies on the CMP project. The 16

ETRM replacement project builds on the functionality being implemented as part of the CMP project, so 17

the CMP project must be completed and stable in production before the ETRM replacement project can 18

begin. The CMP project is scheduled to be completed in 2016.238 19

237 Refer to WP SCE-04, Vol. 2 Bk C pp. 237-242. 238 Refer to testimony for Commodity Management Platform later in Section C – “Power Supply Software

Projects.”

CIT-00-SD-PM-000226 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.00 2.40 - - 5.40 Previous GRC Request - - - - - 6.00 6.00 12.00

Recorded Forecast

182


The existing ETRM system is obsolete and is not being enhanced by the vendor. 2

It runs on older versions of Microsoft Windows Server and Oracle database server, which are no longer 3

supported with software upgrades and security fixes by the vendors. 4

SCE has already selected a new energy trading and contract management 5

platform, which is being used to implement the CMP project. By replacing the existing ETRM system 6

with this new system, SCE can handle conventional, renewable, and financial transactions in a single 7

system. This will simplify the accounting and reporting processes since data from only a single system 8

must be extracted. 9

The existing ETRM system requires a multitude of spreadsheet workarounds to 10

support the current business needs. Moving to the new system will reduce the need for these manual 11

spreadsheet workarounds. 12

(1) Benefits 13

This project will consolidate all of SCE’s wholesale energy trades and 14

bilateral contracts into a single system. It will reduce the need for manual spreadsheet workarounds and 15

streamline the capture and management of energy transactions. It will also simplify accounting, internal, 16

and regulatory reporting processes. 17


We reevaluated the scope of this project as part of this 2018 GRC application, and 19

the total cost estimate was reduced. In addition, we enhanced the cost estimates based on experience 20

gained from the CMP project. 21

The forecast total project costs are $5.4 million.239 The capital forecast for this 22



project includes costs for SCE employees, supplemental workers, and consultants, as well as software 25

and vendor costs. See this project’s workpaper for the cost breakdown information. 26


183



chose not to pursue this project, then we would have to rely on the existing system, which is not being 3

enhanced by the vendor going forward and relies on obsolete operating system and database software. 4

Based on this we concluded that keeping the existing system is not a viable solution. 5


upgrading the existing system to the latest product from the existing vendor to meet the needs of the 7

energy trading and risk management functions. We did not pursue this option because it would not meet 8

our needs. The implementation costs for the latest product would be similar to the new SCE system that 9

is being implemented for the CMP project since the latest vendor product requires a new system 10

implementation rather than an upgrade. In addition, we would have significant integration expenses 11

since half of our portfolio (renewable power) would be in the new SCE system and the other half 12

(conventional power and financial contracts) would be in the latest vendor product. 13


procuring an existing COTS solution to meet the needs of the energy trading and risk management 15

functions. We did not pursue this option because it would not meet our needs. We have already 16

purchased the license for the new system, which is a class-leading product. It is unlikely SCE could find 17

another vendor that can provide better functionality and/or lower price. In addition, implementation 18

expenses for another vendor would be higher than for the new vendor, since for the new vendor we are 19

starting from a system already implemented (through the CMP project) for renewable contracts. We 20

would also have additional integration expenses since half of our portfolio (renewable power) would be 21

in the new system and the other half (conventional power and financial contracts) would be in the 22

second new system. 23

184

7. Aggregated Demand Response (ADR) 1

Table V-55 Aggregated Demand Response240



D.15-11-021 adopted the Aggregated Demand Response project, as part of SCE’s 3

2015 GRC. ADR Phase 1 was developed during 2013-2015 with a total expenditure of $2.29 million, as 4

shown in Table V-55. SCE will complete this project in 2017. 5

Phase 1 was implemented in first quarter 2015. The second phase is scheduled to 6

be implemented in 2017. The project is dependent on the CAISO Proxy Demand Resource (PDR) 7

implementation, which has been delayed. This, in turn, delayed the completion of SCE’s ADR project. 8

The CPUC has encouraged the growth of demand response programs to enhance 9

electric system reliability, reduce power purchases, and protect the environment. In D.05-11-009, the 10

CPUC developed a strategy and initiated programs to promote demand response. In proceeding A.05-11

05-006, the CPUC adopted programs and goals for demand response for SCE and the other California 12

energy Investor Owned Utilities. Demand response constitutes between one and two percent (in terms of 13

MW) of SCE’s portfolio. SCE anticipates that demand response will become a more significant factor in 14

determining overall power requirements going forward. Load aggregators are now allowed to create 15

demand response blocks241 from individual consumers participating in the aggregation programs. An 16

effective demand response program might be equivalent to a significant dispatchable generation 17

resource. 18

240 Refer to WP SCE-04, Vol. 2 Bk C pp. 243-248. 241 Demand response blocks (also known as load control groups or blocks) are aggregations of individual

consumers that can be dispatched (activated) as an all-or-nothing option. For example, a load aggregator may create a demand response block of 2 MW. This block can then be not dispatched (0 MW) or fully dispatched (2 MW), but it cannot be partially dispatched (e.g., 1 MW).

CIT-00-SD-PM-000182 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - 0.02 1.34 0.93 2.59 0.87 - - - 5.75 Previous GRC Request* - - - - 4.15 1.40 0.25 5.80 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

185

In R.07-01-041, the CPUC required demand response programs be integrated into 1

the CAISO markets. Under the market rules of CAISO, the location of the demand response resources 2

will affect SCE’s power costs due to Locational Marginal Pricing, and SCE’s costs for congestion 3

management and Resource Adequacy (RA). SCE must be able to accurately track and model how 4

demand response is affecting SCE’s net power need in order to manage its energy procurement for its 5

customers. 6

The ADR project is implemented in two phases. ADR Phase 1 was implemented 7

in 2015. It enabled Demand Response bidding into the market starting June 17, 2015. 8

The scheduled implementation of ADR Phase 2 in 2017 will provide the 9

additional capabilities to facilitate the full functioning and usage of demand response resources by 10

aggregating the demand resource attributes and providing them to the CAISO Day-Ahead and Real-11

Time markets. 12


In D.14-12-024 the CPUC required SCE to develop a plan for integrating demand 14

response into the CAISO markets by January 1, 2018 to retain the Resource Adequacy value of demand 15

response resources. CAISO requires SCE to provide demand forecast information into its market to 16

mimic generation resources. SCE will require analytical tools to operate in the CAISO market with 17

aggregate demand response as a dispatchable resource. Demand Response resources differ from other 18

energy resources because they bridge the retail and wholesale businesses. Multiple retail accounts (with 19

corresponding meters) that have similar attributes (e.g., location or demand response program 20

participation) are aggregated into a single wholesale resource to be bid into the CAISO market. 21

The Aggregated Demand Response application will provide the following 22

functions: (1) Store the definition of wholesale demand response resources that can be bid into the 23

CAISO market based on an aggregation of retail customers; (2) Forecast the capacity of each of the 24

defined demand response resources on a daily and hourly basis; (3) Generate bids for each demand 25

response resource and submit these bids to the CAISO markets; (4) Retrieve awards from CAISO for the 26

demand response resources and translate these awards into instructions to the retail demand response 27

186

participants;242 (5) Monitor the actual performance of demand response resources by comparing their 1

actual usage during a demand response event to a baseline usage to determine the actual load reduction; 2

and (6) Perform settlements by calculating data (including resource performance data) for submission to 3

CAISO and validate CAISO settlement payments and charges. 4

(1) Benefits 5

This project will comply with R.07-01-041 and will enable SCE’s 6

customer demand response programs to participate in the wholesale CAISO energy markets. Based on 7

the attributes of the various customer demand response programs these programs will be aggregated into 8

wholesale energy resources. These resources will then be bid into to the CAISO Day-Ahead and Real-9

Time energy markets. 10


The total project costs are forecast to be $5.8 million. SCE is requesting $3.5 12

million to implement Phase 2 and complete the project by 2017.243 The capital forecast for this project 13



includes costs for SCE employees, supplemental workers, consultants, as well as software and vendor 16

costs. See this project’s workpaper for the cost breakdown information. 17



modification. We did not pursue this option because it would not meet our needs. If we choose not to 20

implement the remaining functionality, then all the DR programs would not be fully integrated into 21

CAISO. This will introduce operational risks of a partly integrated solution. Potential impacts are 22

suboptimal market participation by SCE and CAISO non-performance penalties. The partial resource 23

integration may also result in Resource Adequacy disallowances. 24

242 This function is complicated by the fact that a bid can be partially awarded by CAISO. For example, if a

demand response resource represents 1,000 retail customers participating in one of SCE’s demand response programs, the bid submitted to CAISO is for 2MW and CAISO awards 1MW to the resource in the auction, the Aggregated Demand Response system needs to determine which retail customers will participate in the program, by how much and for how long.


187


procuring an existing COTS solution to meet the needs of the demand response integration function. We 2

did not pursue this option because it would not meet our needs. SCE evaluated third-party vendor 3

solutions as part of Phase 1 of this project. However, significant product flexibility (and potentially 4

costly custom development) would be required to manage changing CAISO requirements. Also, demand 5

response bidding into a wholesale market like CAISO is a relatively new field with few proven vendor 6

solutions available. SCE therefore developed the ADR system in-house. 7

8. Commodity Management Platform (CMP) 8

Table V-56 Commodity Management Platform (CMP) 244



D.15-11-021 adopted the Commodity Management Platform project (formerly 10

known as the Renewable Contract Management System) as part of SCE’s 2015 GRC. This project 11

implements a new system to manage renewable power contracts and replaces the existing Wholesale 12

Energy System (WES). WES cannot handle the volume and complexity of the new renewable contracts 13

currently online and those that are expected to come online in the next five years in growing numbers. 14


WES was implemented in 1998 and does not have the required functionality 16

needed going forward. Specifically, the current WES system is customized to settle legacy power 17

purchase contracts based solely on whether the power was generated during an on-peak or off-peak 18

period (i.e., using a time-of-use definition). New renewables contract terms require payment and 19

settlement functions based on interval data (hourly or sub-hourly) and are tied to CAISO operations and 20

settlements. Since the WES system cannot handle hourly data, all contracts that require hourly or sub- 21


Recorded ForecastCIT-00-SD-PM-000112 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 1.25 5.63 0.68 0.70 5.49 8.59 2.85 - - - - 25.18 2015 GRC Authorized* 1.25 5.63 0.68 0.70 12.52 7.31 28.08 2015 GRC - SCE Request 1.25 5.63 0.68 5.00 15.52 - - - 28.08 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized our adjusted 2014-2015 request. The 2015 GRC did not address 2010-2012 recorded costs.

188

hourly data (which includes most of the new contracts) are handled manually using spreadsheets, Access 1

databases, and user-developed SAS tools. Data is extracted from WES and other sources, processed 2

manually using spreadsheets and other tools, and finally the results are manually entered back into WES 3

for further processing. This process is cumbersome and increases the risk of human error. 4

In addition, WES was built on technologies that are no longer supported. The 5

limitations of the current system have forced SCE to add more user-developed applications (UDAs) and 6

manual processes to handle the growing volume and complexity of new renewable, combined heat and 7

power, and Qualifying Facilities (QF) contracts. These workarounds cannot handle the growing volume 8

and complexity of contracts, which presents a growing risk of data integrity issues. 9

The Commodity Management Platform, in contrast, can handle hourly (and sub-10

hourly) data through a formula editor that will enable advanced contracts to be configured in the system 11

and automate the processing of these contracts. This will reduce the need for spreadsheet workarounds. 12

Due to the current market structure, the new contracts are far more complex than the legacy contracts 13

that SCE administers. The contracting terms now include, but are not limited to: 14

• Processing for forecast, scheduled, and metered kWh. 15

• Multi-settlement intervals per hour (5, 10, and 15-minute intervals). 16

• CAISO settlement transaction adjustments. 17

• Payment adjustments based on CAISO prices. 18

• Annual energy delivery target monitoring. 19

(1) Benefits 20

The Commodity Management Platform will accommodate contract 21

compliance, payment, and reporting processes for existing and envisioned future power purchase 22

contracts. The Commodity Management Platform will also provide enhanced controls and auditing 23

capability above what can be done using spreadsheets. The system will store a timestamp and user name 24

for all changes to data in the system, and store both the previous and new value of the changed data. 25

This enables users and auditors to trace the source and timing of all data changes. This greatly aids in 26

identifying and diagnosing the source of discrepancies in calculation results and counterparty invoices 27

and provides controls that cannot be replicated using spreadsheets. 28


The total project costs are $25.2 million. The capital forecast for this project was 30


189




SCE has recorded $22.3 million through the end of 2015 and is requesting $2.85 4

million in 2016 to finish the implementation of the project.245 SCE had planned to finish this project in 5

2015, however, we needed to implement new requirements in 2016 associated with a new Formulae 6

Editor for configuring contracts, a new Energy Curtailment work flow, and extension of Integration and 7

User Acceptance testing. Even with these new requirements, SCE plans to implement this project with 8

lower total costs over the 2013 – 2016 period than what the Commission authorized in the 2015 GRC. 9



chose not to pursue this project, then we would need to continue to rely on spreadsheets and manual 12

workarounds to settle current and future renewable contracts. We would also not be able to handle the 13

growing volume and complexity of contracts, which would increase the risk of human error and data 14

integrity issues. 15

Alternative 2: SCE considered custom development to replace the 16

existing WES system. This would be a high-complexity development due to the need to be able to define 17

different settlement calculations for different contract types. Also, a custom development would prevent 18

us from consolidating renewable and conventional power contracts into a single system in the future. 19

9. Generation Management System (GMS) Upgrade 20

Table V-57 Generation Management System (GMS) Upgrade246



Recorded ForecastCIT-00-SD-PM-000149 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - 0.89 0.40 0.98 0.25 - - - - 2.52 2015 GRC Authorized* 0.89 0.80 1.69 2015 GRC - Original Request - - 1.50 0.19 - - - 1.69 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized our original 2014 request, plus the underspent amount in 2013.

190


D.15-11-021 adopted the Generation Management System project as part of 2

SCE’s 2015 GRC. SCE is requesting $0.25 million in 2016 to implement the final phase of the project to 3

be compliant with the NERC CIP V5 requirements. 4

The Generation Management System (GMS) was implemented in 2002 and 5

enables SCE personnel to more accurately determine SCE’s net energy position throughout each 6

operating day, thereby improving the opportunity to lower energy costs to its customers. 7

GMS supports SCE’s goal to minimize costs to its customers by providing real-8

time information for Qualifying Facilities (QF), conventional and renewable power production and 9

enabling SCE to update hour-ahead schedules for generation deviations from planned schedules. GMS 10

functionality allows Power Supply operations personnel to calculate and monitor: Automated 11

Generation Control (AGC); telemetry data; schedules from the Power Costs Inc. (PCI) system; dynamic 12

schedules information; CAISO scheduling information; energy trader information; real time data from 13

various locations; net short positions; and adjustments to hour-ahead schedules based on generation 14

status. The installed GMS software version depends on obsolete versions of Microsoft Windows and 15

Windows Server, which causes maintenance issues. The system was designed and warranted by the 16

vendor for 25 connections to power plants. A software upgrade in 2005 enabled the system to handle 50 17

connections, but is insufficient to handle SCE’s current 105 connections, and projected future portfolio 18

requirements as more renewable generation resources come on-line (an average of 60 new connections 19

are expected each247 year, with 225 total connections by the end of 2018). 20

The new system that has been implemented in June 2016 resolves all of these 21

issues. 22


SCE has recorded $2.27 million through the end of 2015 and is requesting $0.25 24

million in 2016 to finish the final phase of the project.248 The capital forecast for this project was 25



247 Please refer to workpaper CSB-00-P1 in Power Supply (PS) Volume 2 – Energy Procurement for details. 248 Refer to WP SCE-04, Vol. 2 Bk C p. 262.

191



10. Power Supply Projects less than $3 Million 3

Table V-58 Power Supply Projects less than $3M249

Work Breakdown Structure (WBS) Forecasted Capital Expenditures (Nominal $Millions)

Table V-58 above lists capitalized software projects with total project costs less than $3 4

million in capital funding and will be starting and finish within the years 2016 to 2020. Please see the 5

workpapers for these projects for detailed information on each project and its associated costs. 6


WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-DM-DM-000028 Usage Measurement System (UMS) - - 1.20 - - 1.20 CIT-00-SD-PM-000148 PPD Control Systems Hardware Refresh 2.19 - - - - 2.19 CIT-00-SD-PM-000230 Gas Solar Control Systems Refresh 0.40 1.57 0.60 - - 2.57 CIT-00-DM-DM-000079 Work Mgt and Reliability-Centered Maint - - 0.50 1.00 1.00 2.50

Total 2.59 1.57 2.30 1.00 1.00 8.46

192

D. Ethics and Compliance 1

1. Enterprise Content Management 2

Table V-59 Enterprise Content Management250



The Enterprise Content Management project is focused on improving SCE’s 4

capabilities to manage a diverse and complex set of business records. SCE has been in business for more 5

than 125 years. This has created a large volume of business information that supports the Company’s 6

operations. The substantial majority of this information exists in a variety of electronic formats, such as 7

electronic spreadsheets, PDF files, and structured251 IT-supported database systems used across the 8

enterprise. This project will implement a set of eight solutions to minimize the risks of record-keeping 9

non-compliance and provide advanced content management capabilities to improve reliability, safety, 10

and electronic discovery (eDiscovery) operations. The project will also deploy tools and controls to 11

improve the accuracy of records across SCE’s operating units and improve classification of information 12

for SCE to meet its information protection needs. 13

To achieve these objectives, we will implement the following solutions: 14

1. Digital Signatures: Integrate digital signature technology and capability 15

with identified critical business processes that have signature requirements 16

as a part of their workflow. 17

250 Refer to WP SCE-04, Vol. 2 Bk C pp. 286-292. 251 Structured database systems refer to any data that resides in a fixed field within a database.

CIT-00-DM-DM-000068 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.40 5.20 5.40 5.40 19.40 Previous GRC Request - - - - - - - -

Recorded Forecast

193

2. Centralization of Critical Records: Continue to migrate critical 1

records252 from file shares and individual laptops to SCE’s standard 2

content management repository. 3

3. Records Management Enhancements: Enhance records management 4

capabilities for repositories such as SharePoint to comply with records 5

management requirements; apply automation and manage lifecycle of 6

records in alignment with records retention schedule. 7

4. Management of Email Records: Apply records management controls for 8

email records in Microsoft Outlook. 9

5. Automate Records Management: Provide the ability to automate routine 10

records management responsibilities for employees. 11

6. Preserve Digital Records with Extended Retention: Deploy capabilities 12

to preserve and enable access to SCE records that have extended retention 13

timelines ranging from 25 years to 100 years and beyond. 14

7. Enterprise Search: Provide a single user interface for employees to 15

search content across multiple SCE repositories. 16

8. Manage Structured Data Lifecycle: Implement lifecycle management to 17

structured data records to properly store and disposition records according 18

to appropriate procedures. 19

b) Need for Project and Scope 20

As the volume of business data and information grows, and the pace of regulatory 21

and business changes quicken, the importance of having timely, effective, and controlled records 22

management increases. The variety of solutions and platforms where this information is processed and 23

stored is rapidly increasing. This increasingly complex information landscape, combined with the 24

growing pace of change in technology, creates challenges for SCE’s information governance and record-25

keeping capabilities. At the same time, SCE’s operations are highly dependent on this information being 26

252 Critical records include records that, if mismanaged, will affect grid reliability, employee or customer safety,

compliance, Company financials, or the business reputation of SCE.

194

accessible, accurate, and compliant with nearly 5,800 legal citations and regulations.253 Each of the eight 1

initiatives identified below are needed to support the evolving information governance needs and 2

mitigate the associated risks and challenges. 3

• Digital Signatures 4

Need: If a signature is required to confirm an electronic document as a 5

business record, the document must be printed, signed, and then scanned to create the finalized business 6

record. This paper-based signature process is inefficient and expensive, especially when considering the 7

scope of the Company’s operations. In some cases it may be hard to confirm the integrity and reliability 8

of the signature process. In a multi-page document, text and other content can be altered and inserted 9

without being noticed. If the signature process does not have integrity or reliability, the Company may 10

undergo reputational damage, significant financial penalties, and non-compliance events. 11

Scope: This project will implement a digital signature technology, which is 12

supported by standard technology products at SCE. This includes the Microsoft Office suite of 13

applications, including Word, Excel, Outlook, Adobe Acrobat-based documents, and other IT-supported 14

document formats, including specialized computer-aided design and drafting (CADD) document 15

formats. The implementation includes: 16

(1) Digital signatures for all eligible employees, with the ability to digitally 17

sign these documents. 18

(2) Electronic signatures to enhance workflow of critical business processes 19

that involve signatures. 20

• Centralization of Critical Records 21

Need: A significant portion of SCE's critical records are contained in 22

repositories that require enhanced controls such as versioning, legal hold management, metadata 23

tagging, and audit trails. These records are critical to operations, safety, and compliance. To mitigate the 24

risks of mismanagement of these critical business records, these repositories must be migrated to the 25

Company’s enterprise content management system so we can apply proper controls to the business 26

records. These controls provide for advanced security and protection, version control capabilities, 27

253 For example, these regulations and citations include the California Public Utilities Code and Code of Federal

Regulations.

195

enforcement of records retention policies, standardized metadata tagging, advanced search features, 1

legal hold management, and audit trail capabilities. 2

Scope: The centralization of critical records will include performing 3

Redundant, Obsolete, and Trivial (ROT) analysis, deleting redundant and obsolete records that meet the 4

Company’s retention timeline, and migrating records to SharePoint. As part of this migration, the project 5

will establish metadata associated with each record, develop an access-based permission design, and 6

apply retention schedules to manage the lifecycle of critical records of the Company. 7

• Records Management Enhancements 8

Need: SCE’s enterprise content management repositories, ranging from 9

SharePoint to network file shares, serve day-to-day operational needs. But they lack robust records 10

management capabilities that enable records classification, declaration, and lifecycle management in 11

accordance with SCE’s retention schedules. SCE must enhance the existing capabilities of these tools to 12

help verify routine records management operations can be performed. With this solution, SCE can 13

effectuate our records retention policy and enforce legal holds to support pending litigation or 14

investigations. 15

Scope: These records management enhancements will manage content 16

residing within the Company’s repositories, such as SharePoint. This in-place records management 17

capability will provide employees with the ability to properly manage new business information 18

repositories that may be added in the future. This will also provide capabilities to meet the management 19

capabilities found in Department of Defense (DOD) 5015.20 standard industry measure of records. 20

• Management of Email Records 21

Need: Over the past few decades, email has become pervasive across all 22

aspects of the Company’s business operations. The Company produces a significant volume of email 23

every day to support its operations. According to the Association of Records Managers and 24

Administrators (ARMA), an international consortium of leading records management experts, on 25

average 5%254 of all emails are classified as business records. These records must be retained pursuant to 26

the Company’s record retention schedule and given the same records management protections and 27

254 ARMA reference - http://www.arma.org/r1/news/newswire/2016/06/22/report-e-mail-is-greatest-compliance-

risk.

196

controls as document-based records. There is currently no effective solution implemented at SCE to 1

provide for proper records management of email-based records. 2

Scope: This project will implement an email-based records management 3

solution within the Company’s email system (MS Outlook). As part of this scope, the solution will 4

provide capabilities to meet Department of Defense (DOD) 5015.20 standard industry measure of 5

records management capabilities. 6

• Automate Records Management 7

Need: The method for determining records retention qualification (identifying, 8

declaring, and retaining the records) is currently performed manually. This requires time from all SCE 9

employees, and also makes the process vulnerable to unintentional errors and incorrect judgment. 10

Automating our recordkeeping will involve managing the lifecycle of records based on pre-defined rules 11

and automating these rules in record repositories where the records are found. This model is based on 12

pre-establishing rules and using technology to automate how we identify, qualify, and retain records. 13

This continues throughout the entire lifecycle of the records. 14

Scope: This project will implement Rules-Based Recordkeeping (RBR) to 15

enable an automated method of recordkeeping, based on pre-established rules. The solution will 16

leverage pre-established document handling rules, and automate the qualification, declaration, and 17

records retention classification of documents. This solution will collect metadata from the user and 18

apply pre-defined records lifecycle management rules across structured systems like SAP and 19

unstructured repositories such as SharePoint. The objective is to automate records retention rules for 20

about 70 critical record types. 21

• Preserve Digital Records with Extended Retention 22

Need: The Company’s Records Management Program addresses the life-cycle 23

of SCE’s records from creating the record, to retaining it, to disposing of it. Many files have a long 24

retention time and are in electronic format. Over time, the long-term preservation of these records is at 25

risk due to frequent changes in technological electronic formats and media standards, technological 26

obsolescence, and degradability of technological media. Each file format can present challenges in the 27

long-term obsolescence of past format standards. Similarly, the obsolescence and changes to previously 28

standard electronic information storage media presents reliability issues in accessibility of data. 29

The inaccessibility of information related to equipment specifications, 30

engineering drawings, or operational procedures could cause reliability issues in maintaining operations, 31

197

or trigger delays when attempting to restore or resume critical operations. Risks also arise in our ability 1

to produce information related to eDiscovery and other compliance matters if the required 2

documentation is not accessible or retrievable. This solution will mitigate risks associated with 3

managing the long-term preservation of records, which will have significant impacts on SCE's 4

operations, compliance, and ability to support eDiscovery requests. 5

Scope: The digital records preservation capability will preserve on a long-term 6

basis the Company’s records in the face of changes in technological electronic format and media 7

standards, technological obsolescence, and degradability of technological media. This solution will let 8

electronic records with extended retention times remain accessible through a standard electronic format. 9

• Enterprise Search 10

Need: The diversity of electronic information sources has increased greatly, 11

and continues to do so, as the volume of electronic business information has grown exponentially. While 12

the Company has worked to centralize its critical information in its enterprise content management 13

systems, many other business documents exist on diverse sources, such as email, files shares, and local 14

disk drives. The time consumed to retrieve documents or records to support critical operations of the 15

Company is high. Implementing Enterprise Search will significantly reduce the time and cost to retrieve 16

documents/records255 and enhance the Company’s eDiscovery operations. 17

Scope: The Enterprise Search capability will make business information 18

accessible and retrievable from multiple enterprise-type sources, such as enterprise content systems, 19

email, file shares, and cloud-services repositories. The Enterprise Search solution will search and present 20

content from multiple repositories into a single results user interface, thereby providing a consolidated 21

view of this information. This Enterprise Search capability will integrate structured and unstructured 22

information such as engineering standards, specifications and safety checklists, in its results. Users will 23

also be able to filter search results by author, repository grouping, and other pieces of metadata to aid in 24

locating the most current and most relevant content. The Enterprise Search solution will also honor the 25

in-place security protections of the information in its existing repositories. 26

255 Refer to WP SCE-04, Vol. 2C pp. 31-52.

198

• Manage Structured Data Lifecycle 1

Need: Much of the Company’s data is stored in its IT-supported application 2

systems (e.g., SAP). Due to increasing demands and business need for more information, the data in 3

these systems are increasing exponentially. This “structured” information must be properly managed 4

throughout its full data lifecycle, from creation to disposition. 5

We currently have no tool in place to give structured data lifecycle 6

management capabilities. So we manage, archive, and dispose of the data in the Company’s IT-7

supported application systems. Structured data lifecycle management will address issues triggered by the 8

exponential increase in unmanaged structured data and improper retention. This tool will help reduce 9

storage costs associated with archived data and improve performance of day-to-day operational systems. 10

Scope: Unlike the typical lifecycle management solutions based on 11

unstructured data (e.g., Word, Excel, Visio, JPEGs etc.), this effort will enable records management, 12

archiving, and legal disposition of the data residing within the Company’s IT-supported structured data 13

transactional systems, including SAP. 14

c) Cost Forecast 15

SCE is requesting $19.4 million to implement the ECM project. This forecast will 16

implement the scope described above for each of the eight ECM workstreams. We developed our capital 17

forecast for this project using SCE’s internal cost estimation model. This model utilizes industry best 18

practices, references from historical projects, and SCE subject matter expertise to estimate project cost 19

components. SCE’s forecast for this project includes costs for software licenses, vendor labor, system 20

integrator labor, and SCE labor and infrastructure hardware. See this project’s workpaper for the cost 21

breakdown information.256 A forecast for each workstream is presented in Table V-60 below. 22


199

Table V-60 ECM Forecast by Workstream

(Nominal $Millions)


One alternative would be to manage this complex information landscape 2

manually. To manually address the diverse information management needs, we will require significant 3

dedicated staff. This staff will need to augment operational, safety and compliance teams, who will 4

continue to spend additional time and effort to carry out routine tasks associated with searching, 5

declaring, classifying, and dispositioning SCE’s records. This manual alternative could cause high costs 6

over a long period. The distributed employee teams would lack necessary governance compared to a 7

technology solution that enables consistent governance with the help of technology tools. Choosing a 8

manual solution may affect the reliability of company operations, impact regulatory or legal compliance 9

matters, cause unsafe conditions for employees and customers, or may cause financial or reputational 10

damage. 11

ECM Workstream 2017 2018 2019 2020 Total Digital Signatures 0.50 0.50 0.00 0.00 1.00 Centralization of Critical Records 0.00 0.50 0.50 0.50 1.50 Records Management Enhancements 1.50 1.50 1.70 1.10 5.80 Management of Email Records 1.40 1.30 0.30 0.00 3.00 Automate Records Management 0.00 1.40 1.00 1.50 3.90

Preserve Digital Records with Extended Retention

0.00 0.00 0.20 0.30 0.50

Enterprise Search 0.00 0.00 1.00 1.70 2.70 Manage Structured Data Lifecycle 0.00 0.00 0.70 0.30 1.00

Total 3.40 5.20 5.40 5.40 19.40

200

2. Electronic Document Management / Records Management (eDMRM) 1

Table V-61 Electronic Document Management / Records Management (eDMRM) 257



The Commission authorized the Electronic Document Management and Records 3

Management (eDMRM) project in SCE’s 2015 GRC. Since we developed the showing in SCE’s 2015 4

GRC application, several technology advances occurred in the content management industry. These 5

changes have spurred SCE to revise the scope of work for this project and evaluate the necessary 6

capabilities around content management going forward. This testimony will update the Commission as 7

to the revised scope of work for eDMRM, discuss the work completed to date, and detail the remaining 8

work to be done under this reduced scope of work. 9

As described in SCE’s 2015 GRC, eDMRM was originally scoped to serve as 10

SCE’s primary enterprise content management technology. In 2014, however, SCE began an enterprise-11

wide transition to Microsoft Office 365, which replaced our existing IBM Lotus Notes email client with 12

Microsoft Outlook, and added new enterprise content management tools such as Microsoft SharePoint 13

and OneDrive. As a result of this strategic shift, eDMRM is no longer SCE’s primary enterprise content 14

management system. Microsoft SharePoint and OneDrive are now the two main content repositories 15

deployed to SCE employees for this purpose. Due to these changes, the scope of eDMRM will no longer 16

include capabilities for centralizing all Company’s records, migration of self-service for low-risk 17

records, and Lotus Notes email records management. 18

SharePoint records management capabilities in its current state are not mature, 19

and as such, they will be enhanced over the next two to three years. In the interim, limited investment is 20


CIT-00-SD-PM-000217 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - 2.10 6.89 1.42 2.60 - - - - 13.01 Previous GRC Request* - - 2.10 4.70 11.40 8.60 5.80 32.60 *The Commission adopted SCE's request for this project in D.15-11-021.

Recorded Forecast

201

required to migrate some high risk records lacking records management controls into eDMRM, manage 1

Engineering Drawings within eDMRM, and improve system performance. 2

Figure V-12 Content Management Tools


eDMRM houses a large volume of critical records that have been migrated from 4

fileshares over the past few years. Current system performance limitations impact adopting and using 5

eDMRM. Not implementing enhanced records retention and disposition capabilities will affect SCE’s 6

ability to comply with the records retention and disposition requirements. SCE plans to deploy a pilot of 7

the digital signature capability for select business processes across the company. This will help improve 8

the business process associated with obtaining signatures for records that already exist in eDMRM and 9

reduce costs associated with scanning and maintaining original documents with wet signatures. 10


As part of SCE’s 2015 GRC, we originally requested $32.6 million to implement 12

eDMRM over the 2013-2017 timeframe. Since that authorization, SCE has modified eDMRM and 13

expects the total project cost to be $13.01 million over the 2013 – 2016 timeframe. SCE recorded $10.41 14

million from 2013 – 2015 to establish the eDMRM technical infrastructure, migrate around 10 million 15

Public Safety and other high-risk records to eDMRM, implement engineering drawing management 16

unique to eDMRM, and create disposition and retention management capabilities. 17

SCE is requesting $2.6 million in 2016 to complete the project at the revised 18

scope by implementing capabilities previously outlined as part of a multi-phased implementation. These 19

capabilities consist of: improving eDMRM system performance; migrating additional critical records 20

repositories from file shares to eDMRM; enhancing the eDMRM-based engineering drawing 21

Before 2014 2014 and Beyond

eDMRM

MS SharePoint(Office365)

Document Management(Primary)

+Records Management

(Primary)

--Did not Exist--Document Management

(Primary)

+Records Management

(Primary-enhancements required)

Records Management (limited use)

202

management module we implemented in 2015; piloting digital signature technology to streamline 1

manual “wet signature” processes across a small set of organizations within the enterprise; and 2

completing and operationalizing records retention and disposition capabilities within eDMRM. 3

We developed the capital forecast for this project using SCE’s internal cost 4


estimate project cost components. SCE’s forecast for this project includes costs for system integrator 6

labor, SCE labor, and infrastructure hardware. See this project’s workpaper for the cost breakdown 7

information.258 8

SCE’s revised scope and cost forecast for eDMRM represents a reduction of 9

$19.5 million from SCE’s original request. As discussed above, this reduced scope is primarily due to a 10

shift in SCE’s strategy to use Microsoft Office 365 as our enterprise content management platform for 11

managing enterprise content and email. 12


SCE considered forgoing the eDMRM enhancements planned for 2016. 14

This approach has significant compliance implications. Not implementing and operationalizing the 15

eDMRM records retrofit and disposition functionality would prevent SCE from fully effectuating its 16

Records Management Policy, which is based on over 5,800 legal and regulatory compliance 17

requirements. Not enhancing functionality in the engineering drawing module and overall system 18

performance will create a risk in terms of employees being able to efficiently access high-risk 19

documents, while performing critical reliability and business resiliency operations. 20


203

E. Finance Capital Projects 1

1. Plant Ledger Upgrade and Tax Module Installation 2

Table V-62 Plant Ledger Upgrade and Tax Module Installation259



In SCE’s 2015 GRC, the Commission authorized SCE to implement the Tax 4

Department Repairs project. The tax repairs scope of the project was originally scheduled to complete in 5

2014, however during the technical design phase in late 2013, SCE identified dependencies within the 6

PowerPlan suite of capital accounting software that required SCE to align the implementation with an 7

upgrade to the latest commercially available version in order to meet the original project objectives.260 8

The PowerPlan capital accounting software was implemented in 2008 and was due for a refresh given 9

needed updates to the software. Reflecting its increased scope, SCE has renamed this project the Plant 10

Ledger Upgrade and Tax Module Installation project. 11

The upgrade to the PowerPlan capital accounting software version was needed not 12

only to support the full implementation of tax repairs functionality, but also to address maintenance and 13

efficiency impacts associated with the outdated version. SCE originally implemented PowerPlan version 14

10.2.1.2 in 2008, and at this time this version is no longer eligible for standard support from the vendor. 15

This results in more costly custom maintenance support agreements. Moreover, SCE identified over 16

259 Refer to WP SCE-04, Vol. 2 Bk C pp. 300-318 260 In addition to the Tax Repairs module, the eight modules in SCE’s current suite of PowerPlan capital

accounting software that required upgrade are the following: Project Cost Management (work order processing, CWIP, unitization), Asset Management (fixed asset accounting), Depreciation (book depreciation and reserve analysis), Depreciation Studies (actuarial analysis and analysis of book accrual rates), PowerTax (tax depreciation, book/tax reconciliation, deferred taxes), Tax Provision (current/deferred tax provisions), Property Tax (filings, analysis, payments), and Charge Repository (SAP-PowerPlan GL interfaces, allocations, validations).

204

ninety accounting business process inefficiencies that are driven by the aging PowerPlan software 1

version. The accounting business process inefficiencies were focused primarily in the following areas: 2

construction work in progress, property tax, and tax provisions related to depreciation. As a result of tax 3

repairs requirements, software obsolescence, and capital accounting process inefficiencies, SCE 4

proceeded with the PowerPlan upgrade project. From inception through year-end 2015, $5 million was 5

recorded to complete the following project activities: solution design, test installation of the upgraded 6

software, and build of integration with SCE’s SAP system. SCE requests $4.2 million for 2016 capital 7

expenditures to complete solution development, integrated system testing, data conversion, and 8

production implementation. The project will upgrade the PowerPlan application to the latest version to 9

eliminate the need for software customization, improve SCE’s capital accounting processes, and enable 10

new business capabilities to support the company’s property tax and income tax filings. 11


PowerPlan is the software application that manages the sub-ledger for SCE’s 13

fixed asset accounting, property tax and income taxes. As SCE’s business needs for managing capital 14

asset accounting and IRS tax regulations continue to change, PowerPlan capital accounting software 15

must be upgraded to remain technically supported and enable the processes. Since its initial 16

implementation as part of SCE’s Enterprise Resource Planning (ERP) program, however, this 17

application has not been upgraded to a more current version supported by the vendor. Moreover, when 18

PowerPlan was originally deployed, the software did not account for all business scenarios within their 19

product suite. As a result, SCE invested in customizations to enable the processes not fully supported. 20

Over the last eight years, PowerPlan has designed and improved their software to account for many of 21

the SCE customizations as part of their standard product offering. This project will update the software 22

and hardware infrastructure and add application functionality. This upgrade will accomplish the 23

following key objectives: 24

• Reduce risk by ensuring the application has the technology updates needed to 25

mitigate information security exposures. 26

• Reduce application failures resulting from outdated vendor technology. SCE 27

requires custom support maintenance with PowerPlan to resolve issues with 28

the core product and SCE specific customizations. This is costly due to the 29

significant amount of customizations made to the software to meet SCE’s 30

205

business needs. By upgrading to the newer version, SCE can leverage the 1

software’s standard functionalities and support provided by PowerPlan. 2

• Address some of ninety accounting business-process inefficiencies associated 3

with functionality gaps in the software. These gaps were primarily driven by 4

the software being several versions out of date. Since the original 5

implementation, PowerPlan has enhanced the standard product to 6

accommodate the industry’s demand for these functionalities. Through 7

implementation of the PowerPlan Tax Repairs module, the upgrade will 8

enable SCE to maintain a systematic method of accounting, by which the 9

company determines whether expenditures to maintain, replace, or improve 10

transmission and distribution linear property are immediately expensed as a 11

repair for tax purposes, or treated as capitalized assets that depreciate over 12

time for tax purposes. 13


The Plant Ledger Upgrade and Tax Module Installation project will be completed 15

in Q3 2016 with a total forecast cost of $9.2M. The capital forecast for this project was developed using 16

SCE’s internal cost estimation model. This model utilizes industry best practices, references from 17

historical projects and SCE subject matter expertise to estimate project cost components. SCE’s forecast 18

for this project includes costs for software licenses, system integrator labor, and SCE labor and 19

infrastructure hardware. See this project’s work paper for the cost breakdown information.261 The 20

revised scope of the project includes major elements as described above and summarized below: 21

• Upgrade of the PowerPlan application to the latest version - 2015.1.3.0. SCE 22

originally implemented PowerPlan version 10.2.1.2. 23

• Enablement of the Tax Repairs module, which automates the determination as 24

to whether a capital expenditure is deemed a repair for tax purposes and 25

enables SCE to continue to provide customers with the flow-through tax 26


206

benefit associated with the tax-repair deduction in a systematic and auditable 1

manner adhering to IRS guidelines.262 2

• Standard integration of capital asset data across various modules within the 3

PowerPlan capital asset accounting suite, including the Asset Management’s 4

plant ledger and Tax Provision, and reducing manual reconciliations thru 5

spreadsheets. 6

• Standard integration between PowerPlan and SCE’s existing ERP Finance and 7

Enterprise Asset Management (EAM) system, reducing the timeline for 8

month-end accounting close. 9

• Elimination of significant application software customization and accounting 10

process inefficiencies. 11


The following alternatives were considered: 13

Alternative 1: In 2014, SCE evaluated the option not to upgrade the 14

system and lose support from the product vendor due to version non-compliance. Under this option, 15

SCE would assume responsibility for maintenance and enhancement of the application. This option 16

would introduce risk to a critical corporate financial application used for fixed asset accounting and tax 17

filings. Similar to other software products in the SCE portfolio, the application was planned to refresh in 18

the 2016 timeframe. Due to the project timeline and application refresh cycle, upgrading the application 19

as part of this project was the option selected. 20

Alternative 2: In 2014, the SCE team evaluated an option of introducing a 21

new third-party vendor product to manage the tax repairs functionality. This option was not selected 22

because the PowerPlan Tax Repairs module offered this capability and integrated more effectively with 23

SCE’s eight other PowerPlan capital asset accounting modules and existing IT applications. 24


207

2. Corporate Projects less than $3 Million 1

Table V-63 Corporate Projects less than $3M263

Work Breakdown Structure (WBS) Forecasted Capital Expenditures (Nominal $Millions)


three million dollars in capital funding and will start and finishing within the years 2016 to 2020. Please 3

see the workpapers for these projects for detailed information on each project and its associated costs. 4

F. Operational Services Capital Projects 5

1. C-CURE 9000 6

Table V-64 C-CURE 9000 2016-2020 Forecast264



The Commission authorized the C-CURE 9000 project in SCE’s 2015 GRC. SCE 8

is now updating the Commission on the status of this project and the additional expenditures required in 9

this GRC to complete the project. 10


WBS OU Project Description 2016 2017 2018 2019 2020 TotalCIT-00-DM-DM-000071 Legal Legal Re-platform - 2.20 - - - 2.20 CIT-00-DM-DM-000075 Regulatory Affairs Reg Affairs - TM2 Replacement - - - - 1.00 1.00 CIT-00-OP-SM-000025 Finance Integrated Budget Planning 2.80 - - - - 2.80 CIT-00-SD-PM-000216 HR Union Negotiations 2.20 - - - - 2.20

Total 5.00 2.20 - - 1.00 8.20

CIT-00-SD-PM-000139 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - 0.09 3.02 2.03 3.71 0.70 - - - - 9.55 2015 GRC Authorized - 0.09 3.02 - - - - 3.11 2015 GRC - Original Request - 0.09 5.03 - - - - 5.13 * In D.15-11-021, the Commission adopted 2013 recorded costs.

Recorded Forecast

208

Many SCE facilities, including all major buildings, control centers, and 1

generation facilities, are protected by a centrally controlled alarm and event-management system. This 2

physical access control system (PACS) logs all access and attempted access to SCE facilities in 3

connection with our Corporate Security ID badging system. 4

The upgraded PACS system was implemented in 2014 in alignment with 5

requirements described in our 2015 GRC. In 2014, however, FERC authorized the North American 6

Electric Reliability Corporation (NERC) to make the new version of Critical Infrastructure Protection 7

(CIP) Version 5 standards enforceable by early 2016. Because of this new standard and other emerging 8

cybersecurity needs, additional modifications to the PACS application were necessary. SCE decided to 9

use the active project team to perform modification to the PACS in support of these regulatory changes. 10

SCE identified this as an efficient solution to meet the regulatory changes required for the PACS. These 11

modifications increased scope and increased recorded costs by $4.5M. The upgrade will be completed in 12

2016, meeting the NERC CIP V5 and cybersecurity standards. 13


The PACS upgrade project completed the foundational application upgrade 15

aligned with the business drivers described in the 2015 rate case. Due to the change in regulatory 16

requirements and cybersecurity needs, the following requirements were added, leading to increased cost: 17

• NERC CIP v5 expanded its scope to include many additional bulk system 18

substation facilities, necessitating integration with the PACS. Additionally, 19

NERC CIP v5 required enhanced physical access controls at high-impact 20

facilities, which required additional integration with new physical access 21

control devices. 22

• Application redundancy is needed to meet cybersecurity standards for SCE’s 23

critical information systems responsible for the reliable operation of the 24

electric grid. If a disaster situation occurs at one of SCE’s data centers, the 25

PACS should have a redundant technical environment at a geographically 26

separate data center to support continuity of the system, which will provide 27

assurance that physical security controls remain operational. 28


The PACS upgrade project recorded $5 million as part of the original scope 30

described in SCE’s 2015 GRC. Due to the new scope for NERC CIP V5 and cybersecurity, an additional 31

209

$4.5M was required. In 2016, the projected cost is $700,000 to fully decommission the old version of 1

the software and implement the disaster recovery environment.265 2

• NERC CIP v5 – The increased number of in-scope substation facilities and 3

increased security measures for high impact facilities necessitated that the 4

system be redesigned to support the expanded scale. This also includes the 5

integration of new access control devices to support the increased 6

requirements for high impact facilities. 7

• Disaster Recovery – Procure hardware, software, and labor to install 8

redundant application in SCE data centers. This requires configuring the 9

PACS to maintain data integrity through a failover and deploying the 10

infrastructure to perform a failover and failback between data centers. 11


Two other options were considered for the PACS upgrade project due to 13

the new scope introduced in 2014 by NERC: 14

Option 1: Build a second independent environment. This would create 15

operational challenges in maintaining two access control systems. Implementing this option would 16

require the IT and Corporate Security departments to increase staff to operate multiple application 17

environments. Security personnel must use two computing environments to monitor security alarms, 18

increasing operational risks and challenges. This solution was not considered viable because of the 19

operating challenges in coordination, communication, and response management. 20

Option 2: Utilize security officers to provide human monitoring of access 21

controls if the PACS became unavailable. This option was not considered because the vastness of SCE’s 22

territory would pose an adverse impact on the response time to dispatch security personnel to provide 23

access control and monitoring capability. There is significant cost associated with staging a guard force 24

in a state of readiness to respond and ultimately deploying them in an emergency situation. 25


210

2. Operational Services Projects less than $3 Million 1

Table V-65 Operational Services Projects less than $3M266

Work Breakdown Structure (WBS) Forecast Capital Expenditures (Nominal $Millions)


three million dollars in capital funding and will start and finish within the years 2016 to 2020. Please see 3

the workpapers for these projects for detailed information on each project and its associated costs. 4


WBS OU Project Description 2016 2017 2018 2019 2020 TotalCIT-00-OP-SM-000026 Business Resiliency Mobile Field Response - - - 0.74 - 0.74 CIT-00-DM-DM-000065 Business Resiliency Crisis Information Management System - - - - 2.60 2.60 CIT-00-DM-DM-000066 Business Resiliency Seismic Risk Assessment - - 2.00 - - 2.00 CIT-00-DM-DM-000062 Corporate Environmental Services EHSync Environmental Clearance Phase 2 - 0.94 0.37 0.37 0.37 2.05 CIT-00-DM-DM-000059 Corporate Real Estate Facilities Management System 0.35 - - - - 0.35 CIT-00-SD-PM-000214 Corporate Safety Safety Observations 0.62 - - - - 0.62 CIT-00-DM-DM-000074 Supply Chain Management Ariba Deployment and Supplier Portal Dec 1.70 1.00 - - - 2.70

Total 2.67 1.94 2.37 1.11 2.97 11.06