Mike Robinson Cisco

MPLS Architectural Considerations

Thinking Differently About Utility WANS Michael Robinson Director - Connected Energy Services Cisco Systems

2 Cisco Proprietary & Confidential. © 2010 Cisco Systems, Inc. All rights reserved.

Agenda 1. Our Situation � Drivers � Existing Networks � Likely Choices

2. Key Arguments for MPLS � Virtualization � Performance � Security

3. MPLS Architecture Variations � Public vs Private MPLS � IP/MPLS vs MPLS-TP � MPLS extensiveness (how far?)


Drivers that Wide Area Networks Must Address: How will we respond? ;�:

☐ Changing Client (Operations) Requirements � Advanced Applications – Control, safety, monitoring, analytics, workforce enablement, etc � Increases in reliability, performance, data transmission, security, flexibility � Changes in system/application architecture

☐ Evolving Regulatory Requirements � State objectives for system operation � Critical Infrastructure mandates � Governmental objectives for operations � Adoption of National Frameworks (Interoperability Standards) � Cyber- and Physical Security mandates

☐ Complex Enterprise Requirements � Enterprise Applications - voice, video, collaboration, mobility, security (physical and cyber), customer care,

handheld, workforce management � Inadequate assets - marginal transport capabilities, overlay assets, technology refresh, sparing, evolutionary

changes in the communication system rapidly becoming an operational risk � Operational risks - inability to effectively control the network, administration costs, isolated management

systems, forecast increases in workload

Promote collaboration throughout the organization and enable partnerships


Cascading Effects of an Aging Infrastructure and Architecture: Why they’re Not Ideal

: Inadequate � Multiple physical networks rather than multiple virtual networks � Lacks the means of integrating new systems and requirements � Promotes overbuilding purpose-built networks (spiral effect) � Security risks and cumbersome deployment of trust

boundaries � Not strategic for forthcoming operational applications

: Inefficient � 93% Operational Network is TDM-based. “Channelization”

causes inefficient use of resources and premature exhaust. � Purpose-built networks increase clients’ overall cost structure

(assets & expense)

: Unmanageable � Operational complexity in managing multiple point solutions � Non-modular. One large logical domain. Vulnerable to

catastrophic faults.

Key portions of the WAN infrastructure are insufficient for handling core business initiatives

Efficiency Comparison of Packet vs Channelized Approach

Challenges of Siloed Solutions


Existing Substation

ChannelBank

RS232

FXS

RS232 T1

T1

DACS

DFR

Analog Phone

RTUA

RTUB

Gateway

Considerations: 1. Quantity of devices 2. Technologies 3. Administrative responsibilities 4. Degree of intelligence


Future Substation

Switch

Switch Router

RS232

Ethernet

Ethernet

Ethernet

MPLS

RTU

EthernetSwitches

Gateway

IED

IED

IED

RTU

DistributedCompute

EthernetSwitches

IED Phone

IEDIED

Physical Security

HMI

WiFiAP

AMIToP

FAN Radios

Ethernet

Ethernet

Ethernet

RTU

HMI

IED

Considerations: 1. Quantity of devices 2. Technologies 3. Administrative responsibilities 4. Degree of intelligence


WAN Technology Considerations

Pros: • (Generic) Implementation and

management simplicity • Low costs • End-to-end pervasiveness • (Almost) De-facto standard • Standards continues to evolve

Cons: • Lacks stringent controls for

substation communications • (Perceived) Security concerns • Legacy protocol challenges • Single network approach

Layer 3 IP Layer 2 Ethernet MPLS

Pros: • (Generic) Implementation and

management simplicity • Low costs • Pervasiveness (almost) • (Almost) De-facto standard

Cons: • Not meant for expansive WANs • Not best QoS and performance

mechanisms • Security concerns • Ethernet only (utilities need more

interfaces and protocols)

Pros: •Superior technology for expansive WANs •Offers TDM-like features while operating as a packet system •Superior QoS, performance and security mechanisms •Retains policy management within the network

Cons: •Relatively ‘new technology’ for most enterprise users •Layer 2.5 protocol that can make security more complex in some environments

Ethernet

Sub

CC

Sub Sub

Virtual SW

MPLS

Sub

CC

Sub Sub

Any SW

Sub

CC

Sub Sub

IP Any SW

B

C A

B C A

?






Key Arguments for MPLS-based WAN

�Virtualization �Organizational �Operational �Network Evolution

�Performance

�Security


Virtualization is critical for multiple workgroups to have their own networks and optimize their unique

business operations

Data, Operations, & Control Center Networks

Service Provider Network

Gen

Data, Operations, & Control Center Networks Data, Operations, & Control Center Networks Data, Operations, & Control Center Networks

Service Provider Network Service Provider Network

Service Provider

Metering Network

Trans Dist

Utility’s

Metering Network

Organizational

NW

Zo

ne

SW

Zo

ne

SE

Zo

ne

NE

Zo

ne

Performance & Reliability

Z-1

Z-3

Z-5

Z-7

Z-2

Z-4

Z-6

Z-8

L2-Multicast Scheduling

Operational

Functional

Con

trol Z

one

App

Zo

ne

HM

I Zo

ne


Network Evolution Using MPLS

Frame Relay

Frame Relay

Frame Relay

Channel Bank

Channel Bank

Channel Bank

Router

Router

Router

Switch

Switch

Switch

AToM Any Transport

over MPLS

FRoMPLS TDMoMPLS

L2VPN L3VPN


MPLS: More Network Performance Controls MPLS TE Fast Re-Route (FRR)

� Subsecond recovery against node/link failures

� Scalable 1:N protection

� Greater protection granularity

� Cost-effective alternative to 1:1 protection

� Bandwidth protection Primary TE LSP

Backup TE LSP

IP/MPLS

R2

R1

R8


� Massive (44%) packet loss at router B→router E!

The Problem with Shortest-Path

�Changing to A->C->D->E won’t help

Router F

Router C Router D

Router A

Router B

OC-3

OC-3

DS3

DS3

DS3 OC-3

OC-3

� Some links are DS3, some are OC-3

� Router A has 40M of traffic for router F, 40M of traffic for router G

Router E

Router G

Node Next-Hop Cost B B 10 C C 10 D C 20 E B 20 F B 30 G B 30

IP (Mostly) Uses Destination-Based Least-Cost Routing Alternate Path Under Utilized


How MPLS TE Solves the Problem

� Router A sees all links

� Router A computes paths on properties other than just shortest cost; creation of 2 tunnels

� No link oversubscribed!

Router C Router D

OC-3

OC-3

DS3

DS3

DS3 OC-3

OC-3

Router F

Router C Router D

Router G

Router A

Router B

Router E

Node Next-Hop Cost B B 10 C C 10 D C 20 E B 20 F Tunnel 0 30 G Tunnel 1 30


MPLS-FRR Link Protection Operation

� Requires next-hop (NHOP) backup tunnel

� Point of Local Repair (PLR) swaps label and pushes backup label

� Backup terminates on Merge Point (MP) where traffic rejoins primary

� Restoration time expected under ~50 ms

Primary TE LSP

Backup TE LSP

IP/MPLS

R1

25 22

16 22

22

R2 R6 R7

R3

R5


FRR Node Protection Operation � Requires next-next-hop

(NNHOP) backup tunnel

� Point of Local Repair (PLR) swaps next-hop label and pushes backup label

� Backup terminates on Merge Point (MP) where traffic rejoins primary

� Restoration time depends on failure detection time

Primary TE LSP

Backup TE LSP

IP/MPLS

R1

25 36

16 22

36

R2 R6 R7

R3

R4

36

R5


Bandwidth Protection � Backup tunnel with

associated bandwidth capacity

� Backup tunnel may or may not actually signal bandwidth

� PLR will decide best backup to protect primary (nhop/nnhop, backup-bw, class-type, node-protection flag)

Primary TE LSP

Backup TE LSP

IP/MPLS

R1 R2 R6 R7

R3

R4

R5


1. MPLS (to the substation) can be used to – a. Provide Layer 2 access into the substation b. Isolate Layer 3 networks (L3VPNs) from other traffic (see below) c. Create/support unique security frameworks

2. L3VPN security enablement a. Traffic separated from other VPNs (NERC CIP CCA traffic can be in its own closed user group,

Internet access distribution can be its own closed user group) b. Addressing plans separate from and not apparent to other networks and devices c. Constrains IP-spoofing to the originating VPN d. Routes individualized per VRF (critical traffic wouldn’t appear ‘in the wrong locations’) e. Route updates independent between VRFs (deters against malicious attacks) f. Core topology concealed from users g. Through MPLS TE and due to unique core addressing, the impact of DoS can be minimized* h. DoS can be minimized to the MPLS edge and constrained to the originating VPL i. Route limiting can guard against route flood attacks j. Access Control List (ACL) administration simplification k. Enhanced security policy administration achieved using path design along with strategically placed

security appliances l. Inter-VPN routing controlled by centrally administered policies

Enhancing Security using MPLS

* Note importance of PE security hardening and design. CE peers also require ACLs and MD5 authentication.


� CIP-002 Requirement 3 directs the Responsible Entity (RE) to develop a list of Critical Cyber Assets (CCA)1. Also provides criteria to help qualify what is a CCA : �R3.1 Cyber Asset uses a routable protocol to communicate outside the Electronic Security Perimeter �R3.2 Cyber Asset uses a routable protocol within a control center �R3.3 Cyber Asset is dial-up accessible.

� Identifying Critical Cyber Assets2 published by NERC in June 2010 to assist identification of CCA as described in CIP-002 R3. �MPLS is specifically listed as a Layer 2 protocol (NOT “routable”) � If devices in the substation are NOT communicating using a routable protocol, and if MPLS is

transporting the information across the WAN, then the CA is NOT considered a CCA

� In compliance with NERC CIP, MPLS networks provide secure, reliable, flexible and cost-effective communication: � Between CA’s and other smart grid elements that use non-routable (Layer 2) communications � Between CCA’s and other smart grid elements that use routable (Layer 3) communications � Security enhanced when MPLS is configured to establish closed user groups (virtual networks)

•IP packets originating from external endpoints, including the Internet, cannot enter these closed user groups, thus preventing many types of external attacks.

NERC CIP Considerations

1. http://www.nerc.com/files/CIP-002-1.pdf 2 http://www.nerc.com/fileUploads/File/Standards/Critcal%20Cyber%20Asset_approved%20by%20CIPCl%20and%20SC%20for%20Posting%20with%20CIP-002-1,%20CIP-002-2,%20CIP-002-3.pdf

http://www.nerc.com/files/CIP-002-1.pdf






� MPLS-based networks provide secure, reliable, efficient, flexible and cost-effective communication between CCA’s and other smart grid network elements.

� MPLS networks natively support communication with non-routable protocols With the current requirement exemption, Bulk Electric System (BES) endpoints connecting to non-CCA MPLS network (and allowed to be outside the ESP boundary) can communicate over MPLS network with non-routable protocol.

� If/when the non-routable protocol exemption is removed, all communication endpoints providing external access into an ESP, including MPLS endpoints, will be considered CCA (therefore subject to NERC CIP).

� MPLS networks can also be extended for secure communication between CCA’s of the BES and other smart grid network elements in compliance with NERC CIP.

� The traffic isolation capabilities inherent in MPLS provides network-based access control for BES CCA’s and other smart grid network elements.

� Several types of MPLS services can be configured to establish closed user groups.

� IP Packets originating from external endpoints, including the Internet, cannot enter these close user groups, thus preventing many types of external attacks.

NERC CIP Compliant MPLS Design






Comparison of Public vs Private MPLS Models Alternative 1 (Public)

� Monthly circuit budget: �Remote site $500 �Hub/large site1 $2000

� Subsequent network modifications drive internal/external costs

� QoS support drives additional costs

� SLA tough to enforce and non-punitive

� All traffic (from diverse business units) carried in single L3 domain!

Alternative 2 (Private) �Monthly circuit budget:

�Remote site $350 �Hub/large site1 $1000

�Subsequent network modifications are internal costs �QoS doesn’t drive additional costs �SLA enforceable (internally) �Traffic easily segmented into isolated L3 domains for each user group.

(1) Note that Hub site figures can also reflect future telecom cost structure for large/high bandwidth endpoints like cell towers and office locations.


Financial Comparisons ; Private MPLS saves enterprise 47% in monthly telco costs

�Conservative estimates indicate annual telco OPEX savings (or minimizing future spend) from SCADA traffic alone can exceed $1.2M �Taking a holistic approach to diverse and forward-looking transport needs at various sites (Function 1,,,,Function 6, etc), it is rational that private MPLS helps cap OPEX growth for the transport department1 �Savings will need to be vetted with the OPEX for private equipment in the core network

; Private MPLS minimizes future moves/adds/changes costs �Evolving business needs will almost certainly drive such activities

; Faster problem identification and resolution enabled by private MPLS helps minimize future telecom department OPEX

; Faster problem identification and restoration on private MPLS helps maintain uptime for enterprise revenue generation

9 Private MPLS minimizes near-term/long-term telecom OPEX

(1) Private MPLS affords new services to be carried over a single T1 access circuit.


Intrinsic Comparisons ; Private MPLS allows enterprise higher degree of control over end-to-

end service delivery ; Private MPLS offers enterprise business units individual SLAs

�“Pay for what they need” �Telecom processes can be designed around SLAs and KPIs

; Private MPLS supports deterministic traffic �Control traffic and other growing business requirements �T1 and DS3 circuit emulation

; Private MPLS supports multiple Layer 2 services �T1, DS3, Ethernet and switched Ethernet

; Private MPLS utilizes telco transport (circuits) leveraging their workforce/systems at the edge of the network

; Private MPLS can support other enterprise services (Function 1,,,Function 6, etc)

9 Private MPLS enables key service options critical for Enterprise business units


MPLS-TP

Working LSP

PE PE

Protect LSP

NMS for Network

Management Control *

Client node Client node

MPLS-TP LSP (Static or Dynamic)

Pseudowire

Client Signal

with e2e and segment OAM Section Section

*Or Dynamic Control Plane

Connection Oriented, pre-determined working path and protect path Transport Tunnel 1:1 protection, switching triggered by in-band OAM, Options with NMS for static provisioning, or dynamic control plane for routing and signaling

Note: The cloud represents one MPLS-TP network, e.g., it may be in aggregation or access


� Multi-segment pseudowires (MS-PW) enable layer-2/-1 services over a combined MPLS-TP and IP/MPLS infrastructure

� S-PE (switching provided edge router) switches traffic between a static and a dynamic segment

� MPLS-TP domain uses static LSP as PSN tunnel and static PW segment

� IP/MPLS domain uses signaled LSP (LDP or RSVP-TE) as PSN tunnel and signaled PW segment

MPLS-TP MPLS-TP IP/MPLS

Aggregation Access Core Aggregation Access

T-PE S-PE S-PE S-PE

Static PW Static Tunnel

Signaled PW Signaled Tunnel

Static PW Static Tunnel

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27

Primary Control / Data Center Access/ Aggregation Network T&D Substations

Core Network T&D Substations

Secondary Control / Data Center

Data Center

Control Center HMI SCADA FEP EMS Historian Analytics

CPAM VSOM ACS CA LDAP

Data Center

Control Center HMI SCADA FEP EMS Historian Analytics

CPAM VSOM ACS CA LDAP

Multiservice MPLS/IP Core

AAA DHCP DNS

OAM Subsystem

Prime Optical

Prime Performance

Prime Provisioning

Prime Network

NOC

MPLS/IP Ring Aggregation

Bridged (REP) Ring Aggregation

MPLS/IP Linear Aggregation

MPLS/IPoDWDM Optical Network Packet / Hybrid Microwave

Fiber

Substation CE

Substation PE

Substation PE

Substation Switch

Substation PE

Substation PE

Substation PE

Substation PE

Control Center PE

Control Center PE

Control Center CE

Control Center CE

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28

Thank You

Mike Robinson Cisco

Documents

cisco proprietary confidential

new systems

security physical

isolated management

operational network

multiple virtual networks

wide area networks

overlay assets