INOC - Art Meierdirk Presentation from UTC April 15, 2014

© 2013 Utilities Telecom Council

Transforming Critical Infrastructure

The Cost of Modernization: Perspectives on Effectively Managing Today's Advanced Technology

Assets without Breaking the Bank

•  Art Meierdirk, Senior Director, Business Services – INOC •  Dan Belmont, Director, Energy & Utilities - West Monroe Partners •  Gary Sanders, Telecom Manager – Sun Prairie Utilities

Utility & Advanced Technology Overview

Dan Belmont, Director, Energy & Utilities

Utility & Advanced Technology Overview

3

•  Utilities – Drivers for Advanced Technologies •  The Telecom “Tipping Point” •  Advanced Technologies – Applications vs. Telecom Capabilities •  Creating a Roadmap •  Cyber Security and Telecom Synergy •  Lesson’s Learned and Path to Success

Utilities – Drivers for Advanced Technologies

4

•  Mission? •  Over the next 10-15 years, we will transform our delivery

infrastructure into a model we call the “Utility of the Future” which will:

–  Empower customers to control electricity usage and costs

–  Improve service, enhance reliability and cost controls

–  Improve the customer experience

–  Increase efficiency of our operations

–  Enable Distributed Generation

–  Networks and Microgrids

Utilities – Drivers for Advanced Technologies

5

GRID Modernization

End Use Programs

Enabling Infrastructure Investments

Information Technology

Core Telecommunications

Service and Data Center,

Substation, and SCADA

Communications

Transmission, Distribution, and

Substation Automation

(DA/SA)

Advanced Metering

Infrastructure (AMI)

Mission Critical Voice and Data

Systems Backhaul

The Telecom Tipping Point

6

AT&T, Verizon, and LECs announce discontinued analog circuit support in 2013 and service in 2018

NERC CIP version 5

removes the non-

routable protocol

exclusion for critical assets

NISTiR 7628 Smart Grid cyber security standards must

address attacks from disgruntled employees,

industrial espionage, and terrorists ….

95% of Utility companies are going to experience copper theft this year.

Cooper theft caused 456,000 outage minutes, 52 injuries and 35 deaths in one year.

Utilities with AMI have more managed devices per square mile than LECs,

Cable, or Cellular Carriers

Applications vs. Telecom Networks Capabilities

7

Smart Metering & Home Area

Network

Distribution Automation

Substation Protection and

Automation

Network Backhaul

Enterprise Communications

Mobile Voice/Data

Communication Applications New challenges for processes and people

Applications vs. Telecom Networks Capabilities

8

Distribution Automation

Substation Protection and

Automation

§  Coverage of >99% of Meters (preferably Electric, Gas and Water)

§  Higher Latency, Excellent Coverage §  Typically Need Mesh Capability

between Meters and HAN Devices

Network Backhaul

Smart Metering & Home Area

Network

Enterprise Communications

Mid-Tier Backhaul

Mobile Voice/Data

Telecom Backbone

AMI and HAN Communications

Mobile Voice/Data

Applications Communication Systems

Communication Systems Characteristics

8

§  Mission Critical Voice/Data §  Wide Band Mobile Data §  Coverage across Service Territory §  Power Generation Back-up

§  Low Latency Broadband §  Good Uplink Capacity for Video §  Coverage across Service Territory §  Power Generation Back-up

§  Mission Critical Voice/Data §  Wide Band Mobile Data §  Relay Teleprotection Services §  Coverage across Service Territory §  Power Generation Back-up

Applications vs. Telecom Networks Capabilities

9

Tier  OneCore  MPLS

Tier  Three  Access  Collectors

Tier  TwoBackhaul

Explore combinations of Tier 2 solutions with Tier 3 networks to minimize net cost while maintaining Cyber Security and IP design

standards

Creating a Telecom Roadmap

10

Microwave/Fiber  Backhaul    • Backbone  IP  Communica0ons  • Connec0vity  to  AMI  comm.  • Connec0vity  to  substa0ons  • Connec0vity  to  2-­‐Way  Radio  

IT  “Hand-­‐Off”  

ENTERPRISE  BUS  • Customer  Billing/CRM    • Load  Control  Management  System  • Meter  Data  Management  System  • Outage  Management  System  • Distribu0on  Management  System  • Energy  Management  System  

Service Center

AMI    •   AMI  Communica0ons  •   Smart  Meters  •   Distribu0on  Automa0on  

• Power  Quality  Meters    • Vehicular  Radios    • Portable  Radios  

2-­‐Way  Voice  &  Mobile  Data  

•   Microwave  or  Fiber  Based    •   SCADA  to  Substa0ons  •   Communica0ons  to  AMI    •   Relay  Teleprotec0on  services    •   Distribu0on  Automa0on  

SubstaLon  &  DistribuLon    CommunicaLons  

Home  Area  Network  (HAN)    •   Home  Energy  Display  •   Programmable  Thermostats  •   Load  Control  Relays  •   Renewable  Energy  Genera0on  •   PHEV/EV  Smart  Charging  

Cyber Security and Telecommunications Synergy

11

Cyber Security and Telecommunications Synergy

12

 MPLS service with four forwarding classes

§  Reserve 2 Mb/s for voice and 4 Mb/s for video

§  Enforce 14 Mb/s PIR for overall service

§  Allow critical and best-effort traffic to burst up to 14 Mb/s if bandwidth is available

 Less overall bandwidth required §  lower overall cost

 Priority and best-effort traffic are equally well served §  Voice and video do not always consume

all the reserved bandwidth

CIR=0 PIR=4 Mb/s

PIR=max

PIR=max

PIR=2 Mb/s CIR=2 Mb/s

CIR=4 Mb/s

CIR=4 Mb/s

Overall SLA=14 Mb/s

Voice

Video

SCADA FE

Normal Data

Change Management and Training

13

= Aware = Knowledgeable = Expert

Telecom/Cyber Security will

Require Training across the Utility

•  Create a holistic business case and roadmap

•  Design cyber security in from the ground up

•  Tight Quality Control, Quality Assurance, and Audit

•  Carefully Orchestrated Cutover/Migration

•  Business Process Design

•  Organizational Change Management

Lesson’s Learned and Path to Success

REB REBMEB

Ping  MEB  to  verify  power  (From  ALU  

gear)

Verify  in  GS  which  MEB(s)/REB(s)  are  communicating  with

Ping  7705  to  verify  power    (Can  be  done  within  SW  or  with  a  device  on  the  same  subnet/network).    Also  

check  SW  for  downed  equipment  alert

Ping  6855  from  7705  to  verify  power.  (Can  be  done  within  SW  or  a  device  on  the  same  subnet/network).      

Also  check  SW  for  downed  equipment  alert

Ping  7705  from  6855  to  test  connectivity  and  

power  

Check  SAM  to  ensure  MEB  port  is  up  and  active  on  

either  6855  or  7705

Check  success  rates  in  G.S  than  PI  to  identify  other  possible  issues

Still  up?No

Refer  to  site  master  to  verify  what  equipment  is  on  site

If  a  77

05  is  o

n-­‐site

If  6855

 is  on-­‐si

te

MEB

Fail  ping

Then

Pass  ping

Pass  Ping

If  all  else  fails:On-­‐site  visit  by  qualified  and  identified  group  through  process  of  

elimination.    Worst  case  scenario.    (SSN,  BIG,  ALU,  Standby,  MJ,  CED)

Pass  ping

Pass  ping

Alert/Remedy  Ticket  from  SSN  that  MEB  is  

down

Find  out  alerting  tool  for  AC  power  loss,  DC  backup  activated,  Backhaul  issues.    That  way  if  other  steps  “fail”  we  can  

start  a  new  process  flow

Pass  Ping

Fail  Ping

Fail  Ping

Fail  Ping

Yes

Gary Sanders Telecom Manager

The Sun Prairie Utilities Experience

Sun Prairie Utilities

16

•  Who and what we are –  Municipality

–  100+ years old

–  Population of 30,000

–  Electric / Water / Internet / WAN

Municipal Entry Into Telecom

17

•  Approached by Sun Prairie Area School District in 1998

•  They had been quoted a price from the local ILEC for $1.2 million to furnish WAN services to School District

•  Cable company was not interested in furnishing services at the time

•  City of Sun Prairie got involved •  Formed study committee between the three entities

•  They used a map of the city to lay out 10 school sites, 5 city sites and 12 Sun Prairie Utilities sites.

•  The result was a natural overhead loop on our electric distribution poles.

•  At that point, we did an engineering study which was funded by the three Committee members for $5,000 each.

•  Also, at this point our Utility Commission became involved and was very supportive.

•  Obtained CLEC license •  Saving to the SPASD & City exceeds $1M

The Committee Met…

18

Launching Fiber WAN

19

•  Simple P2P roll out for School District , City & Utility sites for SCADA

•  Began offering business solutions •  Evolving expectations / scope •  Transition from ATM to Ethernet •  Engineering, design & support needs •  Partner with NOC provider

–  24x7x365 Support –  Reduced staff & tools (monitoring / ticketing)

costs

•  At the time customers were not happy with services & availability offered by Incumbent providers

•  Temporary solution – FTTH technology had not matured enough at the time. We wanted to offer the community something until we found a FTTH product that we were comfortable with

•  Wireless solution would ultimately complement FTTH technology

Why Wireless?

20

Muni Wireless Hype Cycle

21

Enhanced Fiber WAN

22

•  Growing services and support •  Required outside resource / expertise •  Higher level of expectations from subscribers

Partner with NOC provider –  Network engineering, design &

deployment –  24x7 Support –  Reduced staff & tools (monitoring /

ticketing) costs •  Multi-node 10Gb ring architecture •  Serving MDU’s as way to expand

infrastructure and grow network cost effectively

Lessons Learned

23

Fixed Wireless •  Line of sight / unlicensed spectrum interference •  Technology / hardware life cycles •  Complimentary to fiber infrastructure – Not end-all

solution Fiber •  High demand •  Symmetrical bandwidth •  Reliability •  Scalability

•  GPON •  High demand •  Economic Development •  Smart Grid •  Quality of Life •  Customer Service •  Public Private Partnerships •  Political challenges at local & state levels

2014 FTTX Initiative 2.0

24

The Network Operations Center On-going Support

Art Meierdirk Senior Director, Business Services

On-going Support – Service Level Agreement (SLA) Based

26

•  Define the “product” •  Facilities / power / environmental

•  Transport / switching

•  Connectivity •  Set Service Level Requirements

•  Metrics (alarm / call handling / accuracy)

•  Service availability (uptime)

•  Time to entry

•  Restoration response time (time to action)

•  Quality Assurance metrics & reports

Value Proposition Elements

Value Converted to SLAs

Meeting Business / Communication Expectations

27

•  Build a solution for 24x7 support to meet SLA requirements: •  Action on Monitored Infrastructure Alarms

•  Answer Incoming Calls / Messages

•  Notifications of Clients / Customers / Users

•  Escalation and/or Dispatch for Restoration

•  Time to Restoration

•  Documentation of all Actions for Reporting SLA Performance

Tools Process People Integration

Solution Elements

28

•  Tools – Monitoring, Workflow, Documentation & Reporting

•  Process – Best-in-class (ITIL) / Focus on SLA

•  People – Skill Set to Match Functional Requirements

•  Professional (Consultation) Services – Integration of Tools and Process-based on SLA Requirements

”Structured NOC”

Tools

29

•  Monitoring, Workflow Documentation & Reporting •  Device / EMS / NMS / Dry Contacts – Alarm

Aggregation from all Infrastructure Systems

•  Workflow Management / Ticketing System with Time-stamp as Work Progresses or Escalation of Work is Delayed

•  Portal / Web Access for Customer Access to their Information

Comprehensive (Single) View

30

NOC Monitoring

System (Alarm Aggregation)

Secure Connection

Security Systems

HVAC Systems

EMS / NMS

Devices Ideal: Monitoring System to

Aggregate ALL Alarms into “Single Pane of Glass”

Process

31

•  Process (ITIL) – Documented, Standards-based and Focus on SLA Fulfillment

•  Process Flow Mapping •  Work Instructions for Every Process •  Expected Results •  Escalation when Required •  Closure and Documentation •  Reporting •  Real Need for Best-in-class Process

Development

People

32

Skill Set to Match Functional Requirements •  Service Desk •  Tier 1 Support •  Advanced Support •  Subject Matter Expertise

Quantity

Time

Staf

fing

Wor

k Loa

d

Staffing Limitations: •  Budget Constraints •  Staff Additions Often Lag Growth •  Multiple Responsibilities - Staff for Highest Job

Requirements

Structured NOC

33

ACTIVITY TYPE BY TIER: Tier 1 Tier 2/3PERIODIC REVIEW 3% 10%DOCUMENTATION 2% 3%INCIDENT MANAGEMENT 7% 18%24x7 EVENT MONITORING 39%CALLS/E-MAILS 14% 4%

Tier  1  Activities65%

Tier  2/3  Activities35%

•  Tiered Jobs •  Resource Pool •  Maintain Focus •  Meet SLAs •  Control Costs •  Right Sized” Solution

Knowledge Base / Runbook

34

•  Documentation for all Aspects of the Business

•  Architecture •  Organization •  Process Flow •  Work Instructions

•  Contingencies

•  Business Continuity Plan

Other Considerations

35

Service Turn-Up Team •  Dedicated service turn-up team •  Initial set-up and changes – process and

procedure, ticketing configuration, equipment monitoring, etc.

Systems & Development

•  Continued development of monitoring tools •  Integration with third-party/client systems •  Custom monitoring, reporting

Project Management & NOC Run Book Development

•  Development and documentation of operational support processes for clients

•  Advanced Runbook including comprehensive client-specific process flow diagrams and detailed work instructions for Incident Management, Problem Management and Change Management

Customer Advocacy •  Represents customer’s interests •  Reports, QA, QC programs •  Ongoing reviews with client team

Transforming Critical Infrastructure

Questions / Open Discussion

•  Art Meierdirk, Senior Director, Business Services – INOC •  Dan Belmont, Director, Energy & Utilities - West Monroe Partners •  Gary Sanders, Telecom Manager – Sun Prairie Utilities