Protection Book v2

Created by Dominique Verhulst Edition 2

Teleprotection over

packet networks

Legal notice

i

Copyright 2013, 2014 Alcatel-Lucent. All rights reserved

Alcatel, Lucent, Alcatel-Lucent and the Alcatel-Lucent logo are trademarks of Alcatel-Lucent. The information presented is subject to change without notice. Alcatel-Lucent assumes no responsibility for inaccuracies contained herein. All other trademarks are the property of their respective owners.

Editors notes

ii

Edition 1.2

This edition contains new information gathered since the first edition. More specifically section 3 of chapter 2 has been updated with new user data and illustrations.

New illustrations were added to chapter 4.

Edition 2.0

All animations have been reworked. More data was added from real current dierential and distance protection deployments over IP/MPLS networks. Added references to standards.

A new section was added in chapter 2, covering bandwidth and latency considerations to make when packetizing TDM data

The technical validation testing examples and practical use cases have been grouped in a dedicated section in chapter 2. This allows more flexibility to add new examples in later editions.

A new chapter was added to clarify why IP/MPLS is chosen over MPLS-TP

A new chapter was added to explain the importance of synchronization

Acknowledgements

iii

This book was made possible through the help of numerous colleagues, friends, customers and partners. At this point I would like to thank all the people who have helped create this content, provided inputs, feedback and inspired me to put this document together.

In particular, I would like to extend my thanks to the following people.

Dave Richards, Fai Lam, Benot Lridon, Bram De Valck, Dave Sargent, Peter Merriman, Carl Rajsic, Colenso Van Wyk, Lieven Levrau, Matthew Bocci, Hannu Ahola, Danny Knezevic of Alcatel-Lucent Canada, France, Belgium, UK, South Africa, Finland, Australia

Cory Struth, Clinton Struth of Falling Apple Solutions, Canada,

Patrick Colling, Michael Wener of Creos, Luxembourg,

Campbell Booth, Steven Blair, Federico Coele of Strathclyde University, Scotland,

Andrej Grbing, Siemens, Germany

Fernando Castro, Joan Sans, Dimat-CG, Spain

Sincerely, Dominique Verhulst

Foreword

iv

Over the last few years Alcatel-Lucent has met many people from the power utility and telecommunications industry and have come to the conclusion that people who deal with some of the most mission critical applications in power uti l it ies, such as teleprotection, have l itt le understanding of the telecommunications infrastructure that assures the proper operation of their applications. It is considered that the telecommunications system is always up and has a guaranteed service performance. Likewise, the people with a telecommunications background in general lack a good understanding of the mission critical applications that power utilities need to operate in order to assure 24/7 electrical power to our homes and businesses.

This book is meant to bridge this gap between the telecom and operational parts of the power utilities organizations.

The hope is it can help the telecom people to appreciate the critical character of the teleprotection applications and design better, future proof telecom networks. Conversely, we also hope that this book can help instill confidence with the people who operate the teleprotection applications of the power utilities that IP/MPLS can meet and exceed their requirements.

It can also be useful for service providers to understand the strict constraints of the power utility applications in order to enable service providers to provide the appropriate service SLAs that are required in the power utility market.

The Need to ChangeSince the War of Currents was won by the AC camp in the late 19th century, electrical power generation and distribution systems havent changed much, other than that they have scaled massively. New central power generat ion technologies have been developed since then but the continuing search for sustainable energy has been driving the development of smaller scale distributed power generation.

Power utility companies can no longer maintain the status quo of how they deploy and manage their power grid, there is an urgent need to change the way they run their business, how they control and protect the power grid. The grid has become a lot more dynamic and this has a direct consequence on the telecom network that is needed to control it. Power utilities will need to embrace this change and adopt new technologies to cope with the change.

v

Nicola Tesla is considered to be the father of the AC power system. His patented inventions of the AC induction motor and the transformer were licensed by Westinghouse. This put Tesla in the AC camp in the War of Currents

Chapter 1

Introduction

Power Utilities are managing one of the most critical assets of our times: Electricity.

In order to ensure that the power grid is always on, power uti l i t ies put systems in place that continuously monitor the electrical infrastructure and provide information about the state of the power grid at key points in the network. These systems are designed to help protect the power grid against failures and avoid cascading of problems through the grid. One of the key systems that help power utilities protect their grid is called the Teleprotection system. This interactive book will provide you some insights into this application, how it works and how it works in a telecommunications network.

Power substation switchyard

Power Utility Assets - swipe to scroll through images

Why teleprotection?

Operating an electric grid requires the provision of safeguards. In particular, fault detection and subsequent corrective action are extremely important. Teleprotection is an essential requirement for operating and maintaining a reliable, robust and safe electric grid. This functionality has always been required and its

Section 1

IN THIS SECTION

1. Why is teleprotection required?

2. What dierent systems are there?.

3. Pilot based protection systems

4. Constraints of protection systems

5. Who are the players in teleprotection?

Introduction to teleprotection

7

Illustration 1.1 Power grid protection

Example of how power grids deal with failures - pinch open to enlarge, tap to start animation

importance is magnified when Smart Grid deployments include increasingly diverse sources of electric power that are combined and channeled to increasingly diverse consumers of electric power.

Teleprotection is a term that is often used to describe the systems which will detect faults in the power grid and will activate circuit breakers or reclosers to prevent faults from rippling through the network or to restore power to a part of the grid after an outage.

Teleprotection systems can be classified into two main groups:

1. Standalone protection systems2. Pilot based protection systems

The first group of teleprotection systems are referred to as standalone teleprotection systems (also known as distance protections).

Standalone teleprotection systems are monitoring the high voltage line impedance and in case of a failure (and a resulting change in impedance) determine where the fault is on the line and make an autonomous local decision whether to trip the circuit breaker or not.

The standalone protection system is shown in illustration 1.2

The second group of protection systems are referred to as pilot based protection systems. The dierence between this type of system and the standalone teleprotection systems lies in the fact that pilot based systems use a communications channel to allow the teleprotection relays on either side of the high voltage line to talk to each other and exchange information about what they see on the power grid.

The illustration below shows how pilot based systems work.

8

High Voltage power line

Teleprotection Relay

Circuit breaker

Illustration 1.2 Standalone teleprotection system

Tap on the labels for more details, pinch to open to full screen.

Within the pilot-based Teleprotection systems group, we can also make a distinction between two types of protection systems.

Pilot Based Protection Systems:

1. Distance Protection or Teleprotection Systems2. Current Dierential Protection Systems

The pilot based Teleprotection systems are the ones that are of interest for this book since they rely on a communications

channel between adjacent nodes.Distance protection systems are measuring the high voltage line impedance in order (see illustration 1.4) to determine where the fault is, just like the standalone system, however since they work in pairs to monitor both sides of the line, pilot based Teleprotection systems can make better and faster decisions with regards to tripping their local circuit breaker. There is a possibility that the fault is further upstream or downstream in the grid. Therefore, information from an upstream or downstream neighbors Teleprotection relay can be essential in making the right decision to trip the circuit breaker as close as possible to the fault. Teleprotection systems send commands on an event basis. They can tell their neighbor to

9

Tele- or Distance Protections work on the principle that they measure the impedance of the high voltage line to determine the location of a fault.

Illustration 1.4 Principle of teleprotection

Pilot communications line

Illustration 1.3 Pilot based teleprotection

block or to trip in case they see a fault. These protection schemes are generally referred to as blocking and permissive protection schemes. The animation in illustration 1.4 shows the principles of these protection schemes.

Current Dierential Protection systems work dierently from the Tele- or Distance Protection systems. They use a dierent mechanism to determine if there is a fault condition on the grid. The principle of operation of Current Dierential protection systems is based on a basic rule of electrical systems: Kirchos point rule. This rule says that the sum of the currents flowing into a point is always identical to sum of the currents flowing out of the same point. If there is any dierence, there is a fault.

Current Dierential relays continuously send sample values to e a c h o t h e r t o compare the current values that they see (as in illustration 1.6). The dierential relays rely on a fixed latency time to transmit and receive their data because they need to compare real-t ime

sample values and use a fixed oset to compensate for the communications latency. Hence the need for low latency, accurate timing and very low variation in the latency.

Constraints of Teleprotection systems

A failure such as a short circuit in a high voltage power system can cause major damage that can lead to an avalanche eect on the entire power system of a city, region or country. It is therefore

10

This animation shows the principles of operation of blocking and permissive protection schemes.

Illustration 1.6 Distance protection schemes

Illustration 1.5 Kirchos point rule

essential that failures are detected rapidly and that protection systems can be engaged immediately after a failure is detected.

Todays teleprotection applications are typically developed using a total fault clearing time of five to seven electricity cycles. The electricity cycle is at the rate of 50Hz (20 ms per cycle) in most parts of the world and typically at 60Hz (16.66 ms per cycle) in the Americas. As shown in illustration 1.8, the fault inception (1 in

figure) and fault resolution delay (4 in figure) take between one and three-to-five cycles, respectively. This leaves one cycle or 16.6 ms (20 ms in a 50Hz grid) for total end-to-end delay comprised of teleprotection (TPR) equipment delay (2 in figure) and telecom network delay (3 in figure). So if teleprotection relay delay was found to be around 3 ms at each terminal, this leaves approximately 10 ms for an acceptable total telecom network delay. The 10 ms maximum latency for digital systems is defined

in the IEC 834-1 standard. Older teleprotection systems that are 11

This animation shows the dierent steps in Teleprotection fault clearance and the time associated with each step.

Illustration 1.8 Teleprotection fault clearing

Current dierential relays continuously send and receive sample values to compare currents, if they measure dierent values, the relay is tripped.

Illustration 1.7 Current dierential protection scheme

using analog (voice band) communication interfaces can be allowed up to 15, 20 and even 40 ms in some cases.

Latency is not the only constraint to which the teleprotection application is subjected, the variation of the latency (even if it is still within the overall 10 ms limit) is also a very important factor. Latency variation is also often referred to as Jitter.

For teleprotection applications the Jitter tolerance is less than 1ms, typically 0.5 ms.

Pilot based teleprotection systems rely on a communications channel between them in order to send and receive information. This means that information is sent from both sides to the other side of the protected line. Hence we have two communications paths, one forward and one reverse path. For Teleprotection to work properly it is important that the latency is the same for both forward and reverse paths, the dierence has to be less than 2ms and needs to be constant (within the Jitter tolerance). This is often referred to as path asymmetry, or the dierence between the forward and reverse path.

Another standard that defines delivery times for power substation applications is IEEE 1646. Generally it defines latency for protection applications between 8 and 16 ms.

Who are the players in the teleprotection market?

The vendors who are active in the teleprotection relay market have been there for quite some time. Names such as ABB, Alstom Grid, General Electric, Schneider Electric, SEL, Siemens and ZIV-Dimat are often found in the Power Substations environments.

12

Illustration 1.9 Time constraints of teleprotection

Section 2

Test your knowledge of teleprotection

13

Review 1.1 Teleprotection Principles

Check Answer

Question 1 of 5What is the purpose of teleprotection?

A. To monitor the condition of the power grid

B. To isolate faults in the power grid

C. To prevent damage to critical parts of the power grid

D. All of the above

Chapter 2

Communication Networks for Teleprotection

This chapter will cover the communication technologies that are used in power utility networks and more specifically the ones that are used for real time applications such as teleprotection.

Before we start going into more detail about the communication networks which are used for teleprotection applications, it is worth elaborating a little on the communications interfaces used by teleprotection equipment.

Most older generation teleprotection equipment is using analog voice interfaces. The teleprotection equipment basically sends audio tones across the communications link which could be as basic as simple copper wires. There are dierent frequencies that are used for the guard tone (to signal that the teleprotection relay is operational) and for the commands channels. The frequencies are all within the audible spectrum from 1100Hz up to 4000Hz. The frequencies and modulation schemes are defined by ITU-T standards. Commands are typically sent at very low speed (200 baud or lower) using frequency modulation.

The physical interface used by analog teleprotection equipment is most commonly the 4 wire E&M interface. This interface uses 6 wires; 2 wires for transmit, 2 for receive and 2 for signaling.

More recent teleprotection equipment is using digital interfaces. Over the past 20 years or more, there has been an evolution of these digital interfaces; the most common ones found on teleprotection relays are X.21, G.703, IEEE C37.94 and most recently Ethernet interfaces.

Section 1

IN THIS SECTION

1. Analog interfaces

2. Digital interfaces

Interfaces of teleprotection equipment

15

Guard Tone

Channel

Illustration 2.1 Frequency spectrum of analog teleprotection systems

Teleprotection relays rely on a stable, symmetric, constant delay telecom network. Traditional telecom networking utilizes TDM transmission architectures based on PDH/SONET/SDH to provide the communication channel between relays. The circuit switched nature with fixed frame lengths provided some guarantee of delay limit, delay stability and transmission symmetry. Existing PDH/SONET/SDH

Section 2

IN THIS SECTION

1. Traditional network architectures

2. Limitations of traditional networks

3. Other factors which drive the need to change

Legacy Networks

16

Illustration 2.2 Typical traditional network architecture and frame structure

The trouble with our times is that the future is not what it used to be.

- Paul Valry - French Poet and critic (1871-1945)

networks have proven to be well suited to the task of supporting current dierential protection, which has demanding performance imposed upon the communications networks.

Over the past few years, there has been a growing demand for more Ethernet and IP based services to be provided for power substations. Engineers are expecting to be able to connect to their corporate intranet from within the substation. Analog telephones are being replaced by Voice over IP models, CCTV cameras are moving to IP and multicast based solutions.

But perhaps the most important driver towards Ethernet and IP in the substation is the deployment of IEC 61850 based substation automation systems. These require Ethernet connectivity and hence we see more Ethernet switches and routers being deployed in the power utilitys communications networks.

While recent SONET/SDH equipment does provide Ethernet connectivity, they are ill equipped to deal with the scale and complexity of oering Ethernet and IP services for large power utility networks.

The problem with non-MPLS based IP routing and Ethernet switching technologies however is that they arent well suited to deal with real time data from applications such as SCADA and teleprotection, so other solutions need to be considered.

Besides the l imitations of the existing power uti l i ty communications networks, there are a number of other business related factors that are driving the need to change.

The drive for Power Utilities to transform their telecom and corporate Information and Communication Technology department (ICT) into an integrated operations and ICT organization is the direct result of the need to reduce costs, maintain or increase performance while making the transition to a smart gird enabled company.

17

This animation shows how Power Utility companies are transforming their operational and ICT departments into a converged organization in multiple steps.

Illustration 2.3 Power Utility telecom transformation

This organizational transformation poses some significant challenges for the network infrastructure because the operational telecoms applications are mission critical and have fundamentally dierent requirements in terms of bandwidth, end-to-end-delay and jitter compared to the corporate ICT applications. The latter have been deployed on Ethernet and IP networks for a decade or more while the first have relied on more conventional TDM technologies for several decades. Bringing those two dierent worlds together onto a common telecommunications infrastructure is by no means a trivial task. However, the benefits are significant and the transformation must happen.

Why is this happening and why now?

There are several key drivers forcing this transformation and making it very relevant and timely.

The Smart Grid. There is a big push from governments worldwide towards smart grid capabilities which have a significant impact on how power utilities have to manage demand and supply and be able to more rapidly adjust to both. This results in more sensors and actuators to be deployed producing more data to be collected and transmitted in real time. At the same time, applications using that data are communicating with various systems in the network.

Security Issues. Coupled with government demand for smart grid capabilities is the increase in security directives. These are a result of the increased terrorist and intrusion threats to power infrastructures. These directives are having a direct impact on the network.

18

Technology Lifecycle Issues. Most of the TSO and DSO communications networks have been using TDM technologies for the past 20 years or more. Government regulations such as RoHS and major shifts in technology in the service provider and enterprise markets are making it very dicult for the vendors of TDM technologies to maintain their investment in the development and production of this technology. Most service providers and enterprise customers have evolved their communications networks to other technologies such as Ethernet and IP which resulted in a steady decrease of demand for TDM products over the past decade. Consequently, there is considerably less investment in the development and support of TDM technologies. There are fewer vendors on the market with native TDM products while vendor interoperability and end-to-end management is often a big issue. Along with this aging of technology, maintenance costs are increasing and finding skilled sta to operate and maintain these legacy technologies and equipment is proving to be very dicult.

Leased Line Service Migration. As already mentioned earl ier, service providers are changing their network

infrastructure to packet based technologies in order to reduce their costs, deal with increased bandwidth demand, support new applications and oer new revenue generating services. Service providers therefore are greatly reducing traditional leased line services to the point where they are no longer oering TDM based leased line services in many areas. Power utilities often rely on leased line services either as backup to their own network or to reach remote substations. As a consequence of the service providers decision to no longer oer these services they are

forcing packet services on power utilities.

Own Fiber Optic. Power Utilities have deployed fiber optic cable (for instance through new OPGW Overhead Powerline Ground Wire) through much of their infrastructure which provides considerably more available bandwidth to enable the transition to converged, packed-based networks.

Evolution Of Microwave Technology. Many power utilities have deployed microwave commun ica t ions sys tems to connec t substations to the communications network in areas where fiber or leased line services were not available or because microwave was the

19

The Newbridge MainStreet 3600 was a TDM multiplexer that has been sold from 1987 till 2010. It was the worlds first software managed TDM multiplexer

Illustration 2.4 MainStreet 3600

most cost-eective solution. As well, service providers have been upgrading their mobile backhaul networks from TDM to packet based technologies to keep up with the demand for bandwidth and support of IP. This is driving the development of high capacity and quality of service (QoS) b a s e d m i c r o w a v e communications systems, well suited to the utility and Smart Grid needs.

New Technologies Mature. Viable packet based alternatives to TDM and SDH/SONET technologies are mature now and have been deployed widely. Some vendors have implemented new standards on some of their products that support mission critical power utility applications such as Teleprotection, SCADA, Telemetering and Telecontrol to work flawlessly on an IP/MPLS based packet network. However, all IP/MPLS implementations are not equal so power utilities will have to ensure their needs are met when selecting vendors.

New Standards Evolve. Standards bodies such as IEC are working on standards for packet based intra- and inter-substation automation and telecommunication (IEC61850-90-1).

Cost Reduction. Service providers have moved away from SDH and PDH technologies because Ethernet has become a lot more cost eective. As a consequence, they are oering higher speed packet based services as a replacement for leased line services at much more attractive price points. Power utilities can take advantage of this and will save costs on leased lines and also on their own network equipment by moving to more Ethernet based solutions.

Government Funding. In order to ease the move towards Smart Grids and the use of more green energy, governments are making funds available for research, trials and full scale rollouts.

All of the above is resulting in an accelerated demand from Power Utilities to help them understand the new technologies and methods to transform their networks to modern, converged, reliable and agile infrastructures.

20

Illustration 2.5 Alcatel-Lucent 9500 Microwave Packet Radio

With the advent of IP and Ethernet devices in the power systems, many utilities are migrating their telecom networks to IP/MPLS in order to support next-generation interfaces and to achieve a converged network infrastructure for existing, new as well as smart grid applications. From a telecommunications requirement for teleprotection, it is necessary to continue providing the connectivity between relays with a high quality of service quantified by limits on delay, delay variation and asymmetry. An IP/MPLS network can meet these as well as other key requirements such as reliability, flexibility, manageability and maintainability.

A major concern for utilities is whether the IP/MPLS network can meet the strict latency requirements for protection signals between transmission substations. The doubts over IP/MPLS usually concern the ability to guarantee low latency service. First, IP/MPLS is sometimes and incorrectly perceived as connection-less IP-technology that can provide data transport but only with a best-eort like quality of service (QoS). This is the case for IP only however and in contrast, the MPLS part of IP/MPLS makes the solution connection-oriented and capable of multiple guaranteed quality of service (QoS) levels.

Another concern about IP/MPLS networks, as applied to teleprotection schemes, is the notion that the statistical nature of packet networks will impact the performance of teleprotection relays. This notion has been disproved through extensive testing and implementation and is not applicable when the Utility

Section 3

IN THIS SECTION

1. Misconceptions about IP/MPLS

2. Underlying principles

3. Performance of Teleprotection over IP/MPLS

4. Provisioning and Managing Teleprotection services on an IP/MPLS network

Addressing the challenges with IP/MPLS

21

It is not the strongest of species that survives, not the most intelligent, but the one most responsive to change Charles Darwin English naturalist & geologist(1809-1882

operates a private network based on IP/MPLS and, as with a traditional PDH/SONET/SDH network, suitable care is taken in the engineering and provision of the network.

SONET/SDH networks can be provisioned to provide alternate routes for mission critical trac such as that between teleprotection relays. When operating correctly, the network provides less than 50 ms switchover time. This 50 ms time is the reference in terms of resiliency for any new telecom technology. IP/MPLS network technology provides several dierent fully standardized resiliency mechanisms to provide a switchover time of less than 50 ms. These include end-to-end alternate paths and MPLS Fast ReRoute (FRR). MPLS Fast ReRoute provides many pre-calculated alternate paths that can overcome any failure scenario in less than 50 ms. On top of that Alcatel-Lucent provides Non-Stop Service (NSS) capability to the applications riding over the IP/MPLS network. More details about IP/MPLS and the reasons why this is the best long term solution over for instance MPLS-TP is provided in chapter 4.

The Alcatel-Lucent IP/MPLS network supports teleprotection with many features:

IP/MPLS networks utilize Label Switched Paths (LSP) to ensure that all packets associated with a particular service, such as teleprotection, follow the same path. This is often referred to as strict paths. This ensures that the predetermined latency target is always met.

The packets associated with teleprotection communication can be assigned a high priority to guarantee that the criticality of teleprotection requirements are met and reduced packet delay variation through the network assured.

The Alcatel-Lucent IP/MPLS network supports many synchronization options to ensure that the network is properly synchronized. Since the IP/MPLS routers are synchronized, they can provide a good reference clock to the relays that are connected using serial interfaces by using the Network Clock to generate the Service Clock. Next generation relays connected using Ethernet can also be synchronized since the Alcatel-Lucent

22

The Alcatel-Lucent 7705 SAR-8 is a fully redundant and environmentally hardened IP/MPLS router with eight modular slots.

Illustration 2.6 Alcatel-Lucent 7705 SAR, a teleprotection capable IP/MPLS router

IP/MPLS routers support ITU-T Synchronous Ethernet (SyncE) and IEEE 1588v2 Precision Time Protocol (PTP). See chapter 5 for more details on synchronization.

Alcatel-Lucent IP/MPLS routers natively support commonly used teleprotection interfaces including E&M, RS-232, X.21, G703 and IEEE C37.94. It also supports Ethernet for next generation of Ethernet based relays. To reduce latency, it is advantageous to support a direct interface from the teleprotection relay to the IP/MPLS routers. This eliminates the need for a channel bank and the additional latency that is added.

Underlying principles. An IP/M P L S n e t w o r k s u p p o r t s traditional relays using Circuit Emulation Service. The key d e s i g n c o n s i d e r a t i o n f o r supporting Teleprotection over a packet network is how to minimize latency. For an IP/MPLS network, the telecom network latency for TDM trac over IP/MPLS consists of packetization delay, network

transport delay and jitter buer/depacketization delay.

Packetization delay relates to the process of transforming TDM trac into packet data. For network delay, there is fixed delay based on physical link speed and distances involved and a variable delay depends on the number of hops (nodes) in between. Thereby each node adds transit equipment latency. Jitter buer and de-packetization delay relates to the time

required for a data packet to move out of the jitter buer and to get de-packetized into a TDM stream connect ing to the teleprotection relays.

The majority of the telecom network delay occurs at the edge where low speeds are present. Latency in the core depends on number of nodes traversed. The use of coloring and strict paths can be used to engineer the network to avoid p a t h d i v e r g e n c e a n d asymmetr ical delays. In addition, the IP/MPLS router or a GPS clock can provide a real-time clock pulse that can be

23

This animation shows how serial data is packetized and sent across an IP/MPLS tunnel to the remote location where the jitter buer ensures a smooth play out of the serial data

Illustration 2.7 Packetization delay

used for relays to time-stamp their data and thus ensure accurate data comparison and processing.

Packetization delay is the delay imposed at ingress as the TDM data is packetized. Smaller payload sizes with a higher number of packets per second result in lower packetization delay and lower one-way delay. Larger payload sizes with a lower number of packets per second result in higher packetization delay and higher one-way delay. The packet payload size is configurable.

Network transport delay in the core depends on the number of nodes, distance and communications medium, but results mainly from transmission delays. With IP/MPLS trac engineering, a service such as teleprotection follows a pre-determined path through the network that meets the strict latency requirement.

On play out, a Circuit Emulation Service uses a jitter buer to ensure that received packets are tolerant to packet delay variation (PDV). This ensures the successful de-packetization of the payload back into the TDM stream needed for communication with the teleprotection relay. The smaller the jitter buer, the less delay is imposed. However, the jitter buer needs to be set at a large enough value to ensure that jitter cannot cause a communications failure on the teleprotection relays. The selection of jitter buer size must take into account the size of the TDM-encapsulated packets. Larger payloads will require larger jitter buer sizes. A properly configured jitter buer provides

continuous play-out, thereby avoiding discards due to overruns and underruns.

Another key element to assure the correct functioning of TDM

applications over a packet network is the timing or synchronization. Over the past few years, several techniques have been developed to provide high precision timing over packet networks. The most important ones to consider for the use in teleprotection applications are Synchronous Ethernet and IEEE 1588v2. Both technologies scale well in large networks and provide timing precision that is appropriate for the teleprotection

24

IP/MPLS enables simplification of the network and supports all critical services in a deterministic way.

Illustration 2.8 IP/MPLS network for power utilities

systems. However Synchronous Ethernet does have the advantage that it is a layer 1 based synchronization solution and therefore is totally immune to packet delay variation and it interworks well with SONET/SDH network synchronization. This is an element that can be important in migration scenarios. Alcatel-Lucent IP/MPLS routers support both Synchronous Ethernet and IEEE 1588v2, allowing complete flexibility in the synchronization design of the network.

Next generation relays now utilize Ethernet interfaces. Point-to-point relay communication can be supported with an IP/MPLS network using Ethernet Virtual Leased Line (VLL) service and multipoint communication such as IEC 61850 GOOSE messaging can be supported using Virtual Private LAN Service (VPLS).

Managing an IP/MPLS network for power utility applications. IP/MPLS is proving to be capable of handling the teleprotection trac. However one of the key concerns of power utilities to embrace this technology is the perception that managing IP/MPLS networks, provisioning services and managing alarms is very complex.

Alcatel-Lucent has spent the past ten years developing a suite of network and service management products which leverage the many years of experience in managing PDH, SDH and ATM networks in order to simplify the task of managing IP/MPLS networks. One of the main components of this suite of management products is Alcatel-Lucent 5620 Service Aware

Manager (SAM). This product leverages more than 25 years of know how in network and service management software development. It is building onto the experience of the 5620NM products that were developed to manage the former MainStreet3600 TDM network products.

5620 Service Aware Manager is a client-server based management system that supports up to 50 concurrent clients which can be set up according to specific rules related to span of control and scope of command.

25

5620 Service Aware manager, manages end-to-end services, network elements, all parts of the network layers, synchronization, SLAs and much more.

Illustration 2.9 5620 Service Aware Manager

5620 SAM can be configured in an active/standby server configuration. The switchover happens automatically in case the active server fails, or it can be triggered from the command interface.

5620 SAM allows thousands of nodes to be managed from the server. It allows the configuration of the nodes, creation of the point-to-point, point-to-multipoint and meshed services from an easy to use graphical interface. It monitors services against specific service level agreements (SLA). SAM provides the visualizing of all the paths and the services in the network in the network and can raise an alarm in case a path is re-routed and fall outside the limits set by the service SLA. Furthermore, 5620 SAM has a northbound interface, based on XML that allows the

integration with OSS systems from IBM, HP and others. It also oers the ability to create web based service portals in order to further simplify the day-to-day operation of the network, creating reports etc. To ensure secure communication between 5620 SAM and the OSS system, the XML interface supports encryption.

The Alcatel-Lucent 5620 SAM provides:

Easy-to-use graphical forms for point-and-click element, network and service configuration

Wizards to guide administrators step-by-step through complex tasks

Advanced scripting, templating and rules-based configuration, allowing customization of the Alcatel-Lucent 5620 SAM for specific network or service requirements. This customization also allows non-expert resources to handle more complex tasks and eliminates repetitive data-entry activities

It is not only important to have the tools to manage the elements, services, protocols, it is equally important to be able to plan ahead. For this Alcatel-Lucent oers dierent options. First there is the 5650 Control Plane Assurance Manager which is integrated with the 5620 Service Aware Manager. It allows to perform some o-line simulations. This means that users can look at dierent what if scenarios such as, what if this link fails or what if this node fails. Network modeling an capacity planning can be integrated with third party applications such as these from Aria Networks and Opnet, seamlessly integrated with 5620 SAM through the open XML interface.

26

Illustration 2.10 5620 SAM screenshot samples

Further automation and zero-touch configuration, if required, can be provided by deploying a service portal such as the Service Portal Express for Utilities.

The Alcatel-Lucent Service Portal Express for Utilities is a lightweight, web-based application tightly coupled with the Alcatel-Lucent 5620 Service Aware Manager (SAM). It is purpose-built for utility operators to simplify IP/MPLS network operations and management, delivering utility-specific workflows and terminology to bridge the gap between network experts and operators with less extensive training. For added security and control, workflows may also be routed to proper authorizations for review and approval.

The Alcatel-Lucent Service Portal Express for Utilities enables rapid, cost-eective deployment by providing a framework-based architecture that is both modular and extensible.

Base modules are included to provide the following capabilities:

Provisioning Monitoring Troubleshooting Reporting

27

Illustration 2.11 Service Portal Express for Utilities is making provisioning, monitoring and reporting extremely simple.

Packetization Principles

In the previous section we covered the underlying principles of the packetization technique which is used to take data from a serial (i.e. X.21) or n x 64kb (i.e. E1, G.703, C37.94) interface and send it across an IP/MPLS network. In most cases there is an optical fiber infrastructure available that oers plenty of bandwidth, so there is no immediate need to focus on the bandwidth consumption impact of packetization. However, there are a number of cases where there is no fiber optic network available as is the case when using microwave radio links or copper lines with xDSL modems.

In those cases, it is important to understand the impact of the parameters, used to configure the packetization system, on the bandwidth requirement that results from these parameter settings. Therefore it is worth looking at the principles of packetization in a little more detail.

Packetization is the process that occurs when legacy interface data (serial data or data coming from DS0 channels in a T1/E1) is presented to a routers ingress port. The router needs to take a series of bytes and put them into packets (=encapsulation) at a certain constant rate which then get routed to their destination. This is done according to a standard mechanism as defined in the CESoPSN standard (RFC 5086). An alternative mechanism to transport TDM over

Section 4

IN THIS SECTION

1. Packetization principles

2. Impact of jitter buer and payload size configuration on bandwidth requirement

3. Validation testing

4. Examples of implementations

Bandwidth & Latency considerations

28

a packet network is Structure Agnostic Transport over Packet (SAToP). This method is not taking the DS0 channels into consideration and basically transports the complete E1 or T1 transparently.

The Jitter Buer assures a smooth play-out of the serial data on the egress port and compensates for packet delay variations that may occur during transit of the packet through the network and intermediate network nodes.

When a CESoPSN circuit is provisioned on IP/MPLS routers we will need to define the payload size (in bytes) at the ingress port and the Jitter buer depth (in ms) at the egress port.

The Impact Of Payload And Jitter Buer

The configuration of both parameters will have an immediate impact on the number of packets that will be generated and the bandwidth consumed on the network ports.

Lets consider an example whereby we packetize data from an E1 interface.

The E1 frame structure consists of 32 bytes with each byte representing a dierent time slot of the E1 frame: TimeSlot 0 to TimeSlot 31. With the E1 interface speed being 2.048Mb/s, each E1 frame takes 125 microseconds. The structure of an E1 frame is illustrated in illustration 2.14. The E1 structure consists of cycles of 16 frames which ensure CRC4 error checking. The time

it takes to complete a full E1 cycle (16 frames) becomes 16 x 0,125 ms = 2 ms.

When packetizing data such as that from an E1 or T1 circuit, there are a few key parameters that will have an important influence on the resulting end-to-end latency and the bandwidth requirements. These parameters are the payload size (PS), this is the amount of data (in bytes) we take from the E1 or T1 circuit (or other serial data) to put in each packet. For an E1/T1 circuit this

29

An E1 frame consists of 32 PCM channels or time slots , which means they are 8 bits in length each. Timeslot 0 is used for framing and error correction, timeslot 16 is used for signaling purposes of the voice channels in timeslots 1-15 and 17-31.

Illustration 2.12 The E1 frame structure - tap to play

would be the number of time slots (N) multiplied by the number of frames (F) we want to send per packet. For the purpose of calculating the formula is:

PS = N x F is expressed in bytes

We know that our frame rate (FR) is one frame every 125 microseconds.

With this information we can calculate the packetization delay (PD) by multiplying the frame rate with the number of frames we want to send per packet.

PD = FR x F typically expressed in ms

Every telecom equipment such as a router will introduce a fixed delay, this will be in the range of tens or hundreds of microseconds depending on the architecture of the product.

This fixed delay (FD) needs to be added to the calculation of our total delay.

The next important parameter is the jitter buer size. The concept of the jitter buer has been explained in section 3. The Jitter Buer is needed to assure a smooth flow of data and minimize packet delay variation. The jitter buer size (JB) is a parameter that is configurable, typically in increments of 1ms. When a router starts to play-out the data it receives at 50% of the jitter buer,

the resulting jitter buer delay (JBD) is half of the configured JB value.

The total delay (TD) therefore becomes the sum of the packetization delay, the fixed delay and the jitter buer delay.

TD = PD + FD + JBD

A parameter which will have a crucial impact on the bandwidth utilization is the packet rate (PR) in packets per second, this is the result of the product of the number of E1/T1 frames we chose to put in each packet (F) with the frame rate (FR) which is 0,125 ms or 0,000125s for E1. The formula to calculate the packet rate is

PR = 1/(FR x F) in packets per second

To calculate the resulting bandwidth (BW) in bits per second we need to add the protocol overhead (MPLS, Control Word, Ethernet, ML-PPP,...), which is typically 42 bytes to the payload size (PS) and multiply by the packet rate times eight (eight bits per byte).

BW = (PS+42) x PR x 8 = bandwidth in bits per second

Lets consider the following example.

We are transporting one timeslot of an E1(N=1) and we will put 16 frames in each packet (F=16), therefore the payload size is

PS = N x F = 1 x 16 = 16 bytes

30

The packetization delay becomes

PD = FR x F = 0,125ms x 16 = 2 ms

Lets assume a fixed delay of 0,3 ms and we configure a jitter buer to one ms, then the total delay becomes

TD = PD + FD +JBD = 2 + 0,3 + 0,5 = 2,35 ms

The packet rate in this example would become

PR = 1/(FR x F) = 1/(0,000125 X 16) = 500 packets per second

And so the resulting bandwidth requirement would be

BW = (PS + 42) x PR x 8 = (16 + 42) x 500 x 8 = 232 kbits/s.

As can be seen from these calculations, careful considerations must be made when putting TDM trac over packet networks.

When available bandwidth is constrained for instance over copper lines or microwave links, one must pay attention to the parameters used for the packetization of the data. It is important to understand the boundaries of the application in terms of maximum latency, apply a margin to it and work the numbers back to find the right balance in terms of end-to-end latency and bandwidth requirements.

In networks where the links are built with fiber optic connections, bandwidth is less on an issue and therefore tuning the parameters towards the lowest possible latency is not a problem.

The illustration below shows a sample graph of how the number of E1 frames one choses to put into a single packet has an impact on the bandwidth utilization.

31

Illustration 2.13 Sample table of packetization bandwidth

0

200

400

600

800

32 16 8 4

736

400

232148

band

widt

h in

kb/

s

number of E1 frames per packet

Performance of teleprotection over IP/MPLS

Alcatel-Lucent engaged Iometrix, the networking industrys preeminent testing and certification authority, to test and validate the ability of the Alcatel-Lucent IP/MPLS based 7705 Service Aggregation Router (SAR) and 7750 Service Router (SR) products for implementing an IP/MPLS network to support teleprotection. This testing was done in collaboration with Toshiba who provided their GRL100 relay equipment and engineering personnel who participated in the testing and verified proper performance of the relays. The Toshiba equipment was available in both X.21 interface and Ethernet interface versions, allowing the verification of support for both traditional and next generation teleprotection relays.

Based on a comprehensive battery of tests, it was concluded that a network comprised of Alcatel-Lucent IP/MPLS routers will comply with all the requirements of teleprotection with substantial margin. The IP/MPLS network performed well within the requirements of the teleprotection application that has, to this point, only been supported by circuit switched (TDM) network (e.g. based on SONET/SDH) devices. The Alcatel-Lucent routers support legacy and next-generation device interfaces such as Ethernet and consequently can support both existing teleprotection devices as well as those that will be deployed in the future. This capability is crucial for smooth migrations of utility networks.

Section 5

IN THIS SECTION

1. Performance and validation of Teleprotection over IP/MPLS

2. Examples of implementations

Performance Validation and Use Cases

32

Illustration 2.14 Toshiba GRL-100

If I have seen further it is because I could stand on the shoulders of giants

Isaac Newton English scientist (1643-1727)

In addition to the Iometrix validation, Alcatel-Lucent and third party laboratories have also conducted testing, sponsored by utilities that evaluated teleprotection equipment from ABB, Areva (Alstom) and Siemens with Alcate l-Lucent IP/MPLS communications equipment. Further tests have been conducted with teleprotection equipment from ZIV/DIMAT, the TPU-1 with E&M, X.21 and G.703 interfaces.

These tests again demonstrated that low end-to-end delay could be achieved, that failover capabilities met or exceeded SONET/SDH standards for trac re-routing (< 50 ms) and that total

control of the bandwidth required per application could be achieved with an Alcatel-Lucent IP/MPLS network.

The capability of IP/MPLS networks to support teleprotection is not only proven in the lab, it is proven in actual deployment. Altalink, a Transmission Operator in Alberta, Canada with 11,800 km of lines and more than 300 substations, has successfully deployed an Alcatel-Lucent IP/MPLS network in a live environment since September 2010 supporting teleprotection alongside general utility SCADA and other operational voice and data trac.

More validation tests have happened in November 2012 with Creos, the power utility of Luxembourg. The tests were meant to validate the use of dierential protection relays over the Creos IP/MPLS network. The tests have proven that dierential protection works well over IP/MPLS, using C37.94 and E1 interfaces. The tests were conducted over dierent network paths; once over a short path between two adjacent nodes and a second time over a long path which consisted of 11 routers and a total fiber length of 104km. The end-to-end latency observed during the tests was 3.87ms on the short path between the relays using C37.94 interfaces and 4.56 ms over the long path. With the E1 interfaces on the dierential relays, the end-to-end latency was 3.37ms over the short path and 4.12ms over the long path.

The tests showed consistent results over a longer period of time and prove that IP/MPLS is suitable for dierential protection even

33

1 of 12

The ZIV-DIMAT TPU-1 was tested with E&M, X.21 and G.703 (E1) interfaces.

Illustration 2.15 Teleprotection equipment successfully tested over Alcatel-Lucent IP/MPLS networks

if the data is sent over a large number of hops. Each intermediate hop between the ingress and egress hops adds 77microseconds to the end-to-end latency.

Current dierential protection schemes have been in operation on the IP/MPLS network at Creos since January 2013.

34

Illustration 2.16 Dierential protection testing at Creos

35

These tests all included at least 4 IP/MPLS routers in the path between the Teleprotection equipment, with congestion on the links. (*): Tested on live power utility network between adjacent nodes.

Table 2.5.1 Summary of teleprotection over IP/MPLS performance tests

Vendor Model Type Latency (ms) Jitter(microseconds) Interface

Siemens 7XV Distance 3.12 5 X.21

ABB NSD570 Distance 3.6 5 Ethernet

Areva DIP5000 Distance 3 5 X.21

ZIV/Dimat TPU1 Distance 5.92 E1

ZIV/Dimat TPU1 Distance 8 E&M

Nokia TPS 64 Distance 8.3 G.703 codir

Siemens 7SD52 Differential 3.12 5 X.21

Siemens 7SD52 Differential 3.87(*) C37.94

Siemens 7SD52 Differential 3.37 (*) E1

Areva P541/P591 Differential 3.25 5 X.21

Alstom P545 Differential 3.48 C37.94

Toshiba GRL100 Differential 2.9 3 X.21

Toshiba GRL100 Differential 0.1 3 Ethernet

36

In this particular case multiple T1 links are used from the microwave radio to create an aggregate bundle of 12Mb/s. The protection equipment is GE L90 using RS232 interfaces at 38.4 kb/s.

Illustration 2.17 Current dierential scheme on 138kV over a microwave link - tap to play

The example above also uses microwave links, albeit longer reach. The payload is kept low which increases the packet rate.

Illustration 2.18 Current dierential scheme on 500kV line over microwave links

Examples of implementations

37

The secondary set of protection relays are SEL 311L and are connected over G.703 interfaces at 64kb/s. The jitter buer setting is a bit more narrow, hence the lower latency. The same microwave MLPPP bundle is used as network link.

Illustration 2.19 Same link as the previous example with a separate set of dierential relays

In this example, there are two fiber optic links between the nodes, one is over OC-3 (155Mb/s) and the other is over a 10Gb connection. A second distance protection scheme is used over a diverse path to protect the same HV line. The second set of protection relays is Siemens 7SA522, also connected using G.703 interfaces.

Illustration 2.20 Distance protection scheme of 240kV line over fiber optic links

38

More complex and higher performance current dierential protection schemes are built in ring topologies as shown here.

Illustration 2.21 Four leg current dierential schemeTable 2.5.2 Master ring with E1 interfaces between A and B

E1 Interface

Jitter Buffer

msPayload Bytes SITE B SITE ASITE A Site B

B to A A to B A to B B to A

CESoPSN 5 480 7.33 7.33 7.46 7.46

SAToP 1 64 1.15 1.15 1.26 1.26

Table 2.5.3 Slave ring with E1 interfaces between A and B

E1 Interface

Jitter Buffer

msPayload Bytes SITE B SITE ASITE A Site B

B to A A to B A to B B to A

CESoPSN 5 480 7.46 7.46 7.53 7.53

SAToP 1 64 1.17 1.17 1.29 1.29

39

Table 2.5.4 Master ring with E1 interfaces. Readout of the 7SD52 relay latency in ms

E1 interface

Jitter Buffer

msPayload Bytes SITE B SITE DSITE D SITE CSITE C SITE ASITE A SITE B

B to D D to B D to C C to D C to A A to C A to B B to A

CESoPSN 5 480 5.07 5.07 4.81 4.81 5.31 5.31 5.08 5.08

SAToP 1 64 1.21 1.21 1.11 1.11 1.10 1.10 1.16 1.16

Table 2.5.5 Slave ring with C37.94 interfaces. Readout of the 7SD52 relay latency in ms

C37.94Jitter Buffer

msPayload Bytes SITE B SITE DSITE D SITE CSITE C SITE ASITE A SITE B

B to D D to B D to C C to D C to A A to C A to B B to A

CESoPSN 5 32 4.00 4.00 3.87 3.87 3.93 3.93 3.93 3.93

SAToP 2 32 2.50 2.50 2.37 2.37 2.43 2.43 2.43 2.43

40

Extensive testing has been conducted at the Strathclyde university by Dr. Steven Blair and Dr. Campbell Booth. The full test report is available on the Strathclyde university website: Real Time Teleprotection testing using IP/MPLS over xDSL.

Illustration 2.22 Dierential protection over xDSL with IP/MPLS

This chart illustrates the available bandwidth versus the distance on typical copper lines depending on the DSL technology used.

Illustration 2.23 Bandwidth versus distance for xDSL

Section 6

Test your knowledge on communication networks for Teleprotection

41

Review 2.1 Questions on Chapter 2

Check Answer

Question 1 of 8Place the labels on the image where they fit in terms of interface type and typical speed

E&M

E&M

X.21

X.21

C37.94

C37.94

Chapter 3

Migration Options

In the previous chapter, we have established that IP/MPLS is ideally suited to replace the SONET/SDH based TDM networks in use by power utilities. The challenge is how to seamlessly migrate from SONET/SDH to an IP/MPLS network. In this chapter we will cover a number of possible scenarios.

Migration options will vary depending on a number of factors, most importantly, the availability of bandwidth for the IP/MPLS routers on the communications infrastructure. In cases where unused fiber optic cables are available in the cable bundle that connects the substations, a parallel network can be created which allows the most smooth migration scenario to be used. The picture below illustrates how this could work.

Section 1

IN THIS SECTION

1. When spare optical fiber is available

2. Reuse of SDH/SONET backbone

3. Introducing CWDM optical multiplexing

Migration options

43

Moving from SDH to an IP/MPLS network involves careful planning and depending on the availability of optical fiber, dierent scenarios may be applied.

Illustration 3.1 Using separate fiber optic cables in the same bundle

In cases where there is not extra optical fiber available to create a separate IP/MPLS network, the IP/MPLS network can be built as an overlay on top of the existing SDH/SONET infrastructure. For instance, if there is already an Ethernet over SDH/SONET available at the substation, the IP/MPLS router can be connected to that interface and immediately replace the router connected to it at the substation. If there is no Ethernet interface available on the SDH/SONET multiplexer, then the IP/MPLS router can be connected through an STM-1 interface for instance. This would enable IP and Ethernet services to be deployed at the substation.

Following that, other services like SCADA and Teleprotection can be migrated across to the IP/MPLS platform.

Once this is completed, the TDM/PDH layer can be removed, leaving the SDH/SONET multiplexer as the only legacy network element in the network. In order to remove this SDH/SONET element from the network gracefully, it is best to introduce a WDM layer in the optical network. In many cases there is an existing WDM infrastructure. Alcatel-Lucent oers a full range of DWDM and CWDM solutions. The 1830PSS family of products oers very advanced DWDM capabilities and is also fully integrated within the Alcatel-Lucent 5620 Service Aware Management solution. Furthermore, the 7705 Service Aggregation Router product family oers an integrated (passive)

44

Illustration 3.2 Scenario 2, re-use of SONET/SDH

Illustration 3.3 Introducing WDM as a migration enabler

CWDM solution which can further simplify the migration process. The illustration below shows how a 7705 SAR can be used initially as a passive CWDM solution to initially provide more optical cable capacity before the IP/MPLS capability is added.

There are several CWDM cards available for the 7705 SAR products. The illustrations on the right show a couple of these cards and how they can be used. The 4 channel card also supports the 1310nm wavelength which is typically used in SDH/SONET networks. This makes the 7705 SAR ideally suited to integrate with existing SONET/SDH networks.

45

Illustration 3.4 Implementing CWDM with the 7705 SAR

Single color Passive CWDM Optical Add/Drop Multiplexer card (OADM) for 7705 SAR-8 and 7705 SAR-18.

Illustration 3.6 7705 SAR CWDM modules

Illustration 3.5 Sample CWDM setup with Alcatel-Lucent 7705 SAR

7705 SAR-8/18

7705 SAR-8

7705 SAR-M

7705 SAR-8

7705 SAR-8 7705 SAR-8 7705 SAR-8

Chapter 4

MPLS Technologies for Teleprotection

In this chapter we analyze the dierent options for next generation packet networks to support mission critical applications such as Teleprotection.

This chapter is meant to help power utilities to understand which technology is most appropriate for their needs and to help understand the basic dierences between the proven technology of IP/MPLS and more recent MPLS-TP.

IP/MPLS, contrary to what many people believe, is not a new technology. Its development started in the mid 1990s in order to improve the performance of routers. The idea of using labels instead of IP addresses was driven by the limited performance of routers to perform address lookups and their abilities to scale in large networks. Since those days, the paradigm has shifted, router performance no longer is the prime issue, however network and service scaling is.

The telecommunications industry has been focusing on IP/MPLS since the year 2000 and has created many standards ensuring a fully featured and interoperable framework. IP/MPLS has become the standard communications solution for service providers worldwide and has found great success in mission-critical networks in such industries as rail, electrical utilities, government and public safety organization, defense and some large enterprises.

Some communications vendors are in the early stage of MPLS-TP consideration and implementation as those standards become more formalized. Before any significant uptake occurs though, two crucial questions need answering:

Is it a viable technology?

Is it a compelling technology?

These are even more significant for non-carrier, mission-critical networks for organizations such as power utilities. Initially, the principle driver for MPLS-TP was to apply a simplified subset of IP/MPLS to bridge the gap between the packet and transport worlds by combining the packet eciency, multi-service capabilities and carrier-grade features of IP/MPLS with the transport reliability and OAM tools traditionally found in SONET/SDH. This seems like a reasonable approach on the surface but as you will see, there are a number of drawbacks and limitations associated with this simplified subset. In fact many of the objectives of MPLS-TP have meanwhile been achieved by IP/MPLS technology implementations.

Section 1

Market evolution of IP/MPLS and MPLS-TP

47

Alcatel-Lucent has already achieved the objectives of MPLS-TP through its extensive IP/MPLS product portfolio and management by the extensively proven 5620 Service Aware Manager (SAM) network management tool. This end-to-end management tool supports not only the Alcatel-Lucent IP/MPLS portfolio of routers and switches but also the DWDM long-haul optical transport products (1830 PSS) and packet microwave radio products (9500 MPR) with extensive capabilities and time saving, error preventing solutions.

The next sections in this chapter will describe the fundamental distinctions between IP/MPLS and MPLS-TP, as it pertains to mission-critical industries such as power utilities.

48

The concept of a unified communications framework based around IP/MPLS consists primarily of two main functions. These functions are

Transport Layer Functions

Services Layer Functions

There is a wide variety of transport layer functions used today in the MPLS context, including LDP, RSVP-TE, BGP labels and MPLS-TP.

For the purpose of transport, it is really a matter of comparing the applicability of RSVP-TE and LDP to static MPLS-TP as a transport mechanism linking to the service layer functions. This service layer function, driven by applications, is converging on IP. This is true in the railway metro and mainline market segments, for example, even though some TDM based applications remain (GSM-R, Interlocking, SCADA, operational telephony...). The same applies to utilities, where the applications are rapidly converging on IP (IEC104, Goose, eSCADA...) but where some TDM applications will remain for a long time (Teleprotection, SCADA...).

IP/MPLS

There are two major control plane protocols that allow the creation of an IP/MPLS service network, which are RSVP-TE and LDP. RSVP-TE provides many tools to achieve the same level of control on the network as an SDH network, where LDP provides a much more dynamic environment more equivalent to a full IP network (with IP network we mean typically a connectionless IP network). It is important to note that both technologies can easily co-exist on a single network and that it is at the design stage that one or the other protocol will be used to answer the needs of the respective applications in terms of resiliency and control. This flexibility may depend upon the vendors implementation in their respective products.

The development of RSVP-TE based trac engineering has been ongoing for more than 15 years and is still evolving today to meet the needs of ever converging applications. This technology is widely deployed and very successful in meeting high availability demands. This includes the functionality to facilitate Dynamic Control Planes for automatic bandwidth adjustments and re-routing or optimization, as well as protection and rerouting

Section 2

MPLS Technologies

49

mechanisms. The control plane used for facilitating these functions is indirectly dependent on the presence of an IP based control plane. The IGP will update the trac engineering database (TED), with all the CSPF (Constrained Shortest Path First), link coloring information etc and RSVP-TE will then utilize the information in the TED. This architecture creates an abstraction between the IGP and IP/MPLS meaning that a failure in the IGP will not cause RSVP-TE based paths to fail.

It is within this market that Alcatel-Lucent has pioneered the adoption of the use of this IP based control plane to make use of RSVP-TE and LDP based transport mechanisms to facilitate the establishment of a framework for delivering both Point to Point and Point to Multipoint L2 Services. In addition to the use of these mechanisms, Alcatel-Lucent has also been able to enhance the L2 point to point and multipoint services with L3, L4 and application level intelligence. This allows critical networks to continue to evolve the services on the installed platforms as the industry demands.

This flexibility that ALU has brought to the market is unprecedented and fundamentally based on the in-house based data path hardware and software developments allowing the continuous adoption of new technologies and standards as the industry evolves. Only this approach can ensure investment protection for periods exceeding 10 years.

IP/MPLS For Mission-Critical Networks

A key enabler for the safe and ecient transformation of power utility communications network is a modern, reliable and flexible infrastructure that forms the core network in order to route the critical application trac such as Teleprotection, grid monitoring, control and status and key corporate data eectively, eciently and on time. Furthermore, non-critical applications around substation automation services such as voice, video surveillance,

corporate LAN/ or VPN and even public internet, are also able to leverage the same infrastructure. These types of requirements are forcing many industries, governments and enterprises to

50

The Multi Protocol nature of IP/MPLS means that it can run on any transport network, over any media and it natively supports any protocol on top of it.

Illustration 4.1 IP/MPLS supports all protocols and any transport mechanism

consider an evolution of their communications infrastructures that would be very dierent from their traditional Time Division Multiplexing (TDM) centric networks. A flexible transformation is required to preserve existing investments and to minimize risks. Alcatel-Lucent IP/MPLS communications infrastructure incorporates state-of-the-art technologies to enable an industry, government or enterprise to deploy a future-proof, highly available IP network to continue supporting existing TDM and legacy applications while providing a smooth migration path to IP, Ethernet and IP/MPLS-based services. This new IP/MPLS infrastructure will maximize the cost-eectiveness and eciency of the network without jeopardizing reliability, while enabling the deployment of new devices and applications that can improve operational and workflow eciency.

A highly available IP/MPLS communications infrastructure is ideally suited to support both mission-critical operations and corporate communications requirements. In addition, the Alcatel-Lucent network and service aware management platform allows organizations to improve their eciency by automating and simplifying operations management for communications services, thus reducing the barrier in introducing IP/MPLS-based technologies and services.

MPLS-TP

The Internet Engineering Task Force ( IETF) and the Telecommunication Standardization Sector of the International

Telecommunication Union (ITU-T) have undertaken a joint eort to standardize a new transport profile (TP) for the multi-protocol label switching (MPLS) technology that is intended to provide the basis or the next generation packet transport network. The fundamental idea of this activity is to extend MPLS where necessary with Operations, Administration and Maintenance (OAM) tools that are widely applied in existing transport network technologies such as SONET/SDH or OTN and to enable it to operate in the absence of a dynamic IP-based control plane.

MPLS-TP, as defined in RFC5921 in the IETF, is positioned to enable MPLS to be deployed in a transport network and operated in a similar manner to that of existing transport technologies. It enables MPLS to support packet transport services with a similar degree of predictability, reliability and OAM to that found in existing transport networks. It defines a subset of MPLS protocols for Layer 1 (L1) and Layer 2 (L2) only.

Some of the rationale behind the push for MPLS-TP is that there is an increasing demand for legacy transport networks to support packet based services and hence the need for evolution to MPLS like behavior, as an evolution of SDH. IP/MPLS networks are perceived as complex by some in the transport world and there is sometimes a further requirement to keep the layer 2 and layer 3 networks separate. In many cases the existing SDH infrastructure and transport switches lack native IP support. The alternative to providing native IP support is to combine the architectural,

51

management and operational models of Circuit Switched transport networks with Packet switching optimizations using an MPLS data plane and additional OAM and Protection capabilities.

The main arguments that are generally used to promote MPLS-TP are the following:

Transport solution leveraging SDH experience and therefore with no control plane, leading to a fully manual provisioning

High performance set of OAM and protection tools to allow fast failover times

Capacity to integrate legacy trac inherited from SDH evolution.

Bi-directional transport of data in LSPs reducing the risk of path mismatch between a source and a destination and its return path.

Reduced cost of equipment (CAPEX), as the technology requires less intelligence due to the absence of a control plane.

High level of security due to the absence of IP based control plane.

These arguments need to be put in perspective when comparing to IP/MPLS technology and implementations, but also when determining the applicability of MPLS-TP for industry

applications. In short, these are the counter-arguments for IP/MPLS.

IP/MPLS can deliver very simple transport solutions just as well, however they oer the advantage of allowing very rapid and flexible changes (adding nodes, links, applications) which is a lot more cumbersome with SDH and MPLS-TP.

Alcatel-Lucent oers even more OAM tools as part of its IP/MPLS solution not only to assure fast failover times in the transport layer, but also to assure the services at higher layers.

The capacity to integrate legacy trac is no dierent in MPLS-TP from IP/MPLS, both have CESoPSN and SAToP capabilities.

The bi-directional LSP behavior is guaranteed by Alcatel-Lucents 5620SAM management tool, so there is no divergence of forward and reverse LSP paths possible.

The idea that MPLS-TP reduces CAPAEX is a major mistake because it is forcing another layer 3 overlay to be added to the network which is not the case with IP/MPLS.

The control plane protocols used in IP/MPLS are not part of the MPLS labeled trac for the applications, they are secured and authenticated. Furthermore control plane protection mechanisms are in place to prevent any control plane attack to have any impact on the user/application trac.

52

Building a mixed network with IP/MPLS and MPLS-TP technologies

The relationship between IP/MPLS and MPLS-TP can be depicted in this diagram (used at the MPLS World Congress) where MPLS-TP was promoted for large access/aggregation networks for transport scalability while IP/MPLS remains in edge and core for services, however interest for MPLS-TP at MPLS World Congress has subsided.

The first information that can be extracted from this drawing is that carriers consider MPLS-TP as a backhaul technology, but

still consider that all the services part is to be provided by IP/MPLS. Whether this vision applies to private networks which are typically smaller than large carrier networks is very questionable. Are private industry organizations willing to train two teams on two dierent technologies, quite possibly from two dierent vendors if not necessary? Such a choice has a very high OPEX cost and probably does not apply in markets other than mobile operators.

From a technical perspective, there are also several challenges when interworking IP/MPLS and MPLS-TP domains. The first is that there is no control plane interworking between the two.

53

MPLS-TP is being positioned as the aggregation network for large carrier networks.

Illustration 4.3 End to end vision of large carrier networks including MPLS-TP

Providing end-to-interworking between MPLS-TP and IP/MPLS is quite challenging because of the dierent control plane architectures, disjoint protection mechanisms and dierent Operations Administration & Maintenance (OAM) tools.

Illustration 4.2 End-to-end interworking challenge

The second is that end-to-end interworking is very dicult in a multi-vendor environment and no mechanisms to ensure end-to-end resilience interworking as shown in Illustration 4.3.

Note also that from an operations perspective, there is little chance to find an end-to-end network management tool because MPLS-TP vendors often do not have IP/MPLS solutions and IP/MPLS vendors which may have MPLS-TP solutions generally have two dierent management solutions.

Alcatel-Lucent however have both IP/MPLS and MPLS-TP solutions, managed by the same management system, 5620 Service Aware Manager.

54

The 5620 SAM manages both IP/MPLS and MPLS-TP.

Illustration 4.4 Screen shot of 5620 SAM

This section will review the positioning used to promote MPLS-TP into certain markets, demonstrate how IP/MPLS can meet at least the same level of functionality of MPLS-TP and review other values of IP/MPLS that cannot be met by MPLS-TP.

This section will also describe how Alcatel-Lucent has enhanced the IP/MPLS standard to provide a unique, secure, complete end-to-end solution.

Comparison between MPLS-TP and IP/MPLS technology

Manual provisioning: Some applications like teleprotection in utilities and SCADA in many dierent industries, built on the blue/red model, require the avoidance of common mode failure scenarios. For that matter, in the case where a single physical network is built (typically a ring), there is a need to control the trac between the source (SCADA RTU) and destination (SCADA MASTER), in order to ensure that they dont share a common path or equipment. MPLS-TP being built around manual provisioning can provide this feature, but, IP/MPLS with the use of RSVP-TE (trac engineering), can also provide exactly the same feature. However, the provisioning process is

easier with IP/MPLS due to the fact the provisioning can be static end-to-end as in MPLS-TP but can also be a mix of strict and loose therefore alleviating the load of provisioning required. This solution has been validated by several teleprotection and SCADA vendors.

High performance set of OAM: IP/MPLS has many OAM features that are similar to that of MPLS-TP. The only dierence is in the way it is used, but overall, it provides the same services. The OAM trigger resiliency features available as part of IP/MPLS (Fast ReRoute, primary/secondary LSP, active/standby pseudowire) can also be used to control SLAs. Table 4.4.1 below lists all OAM tools which are available on the Alcatel-Lucent IP/MPLS platforms. Bi-Directional Forwarding Detection (BFD) can be used to trigger fast failover of L3 protocols such as RSVP, VRRP at 10ms intervals.

Capacity to integrate legacy trac: The legacy transport (typically E1) is part of the MPLS standard (CESoPSN and SAToP) and is therefore supported by IP/MPLS and MPLS-TP. Main mobile operators are successfully transporting E1 trac of GSM over IP/MPLS for more than 5 years, demonstrating the

Section 3

Comparing technologies

55

technology is totally suited for such legacy transport.

Bi-directional transport of data in LSPs: In order to totally eliminate the risk, for applications that are sensitive, IP/MPLS RSVP-TE supports static provisioning of paths which allows control of trac end-to-end and ensures that bi-directional trac takes the same path, exactly as MPLS-TP does. Furthermore Alcatel-Lucent 5620SAM will always provision symmetrical LSP paths.

High level of security: IP/MPLS has been deployed by carriers for years and has been demonstrated as a very solid technology to protect data. However, as with all security frameworks, it is not only a matter of technology, but also a matter of process and implementation, to ensure full security. The utility market have rolled out a specific security framework called NERC-CIP (in North America) to protect their mission-critical applications. It is important to note that many utilities networks world-wide have successfully met these criteria with an IP/MPLS network. Many critical Defense networks are using an IP/MPLS infrastructure and have so for years.

Platform cost: Is an IP/MPLS platform more expensive than an MPLS-TP platform? Both can use the same kind of silicon technology, so, there is no reason for a price dierence. Therefore, this question needs to be looked at in the broader context of network requirements on the platform itself, not only in the context of technology. For example:

56

Table 4.3.1 OAM tools on Alcatel-Lucent IP/MPLS routersOAM function

ICMP & ICMPv6 Ping Traceroute

Two Way Active Measurement TWAMP

LSP diagnostics LSP Ping, LSP traceroute

SDP diagnostics SDP ping, SDP MTU path discovery

VLL diagnostics VCCV ping, VCCV trace

VPLS MAC diagnostics MAC ping, MAC trace, MAC populate, MAC purge, CPE ping

Ethernet OAM 802.1ag, Y.1731, 802.3ah

Ethernet loopbacks Line, internal

OAM propagation to attachment circuits

ATM ports, E1 ports, Ethernet ports, Pseudowire status signaling propagation

LDP signaling

Multicast debugging Multicast trace, Multicast stats, Multicast route info

LDP Tree trace Allows to see multiple paths for a given service

Service Assurance Agent SLA monitoring

Control Plane Assurance Management

OSPF, ISIS, BGP, PIM,... network control plane visualization

BFD Bi-directional Forwarding Detection

Is the capability to absorb trac burst to minimize packet loss crucial to the operational applications in mission-critical networks? If yes, then the memory cost would be the same on both platforms.

Is supporting a large number of OAM sessions (e.g. Y.1731 continuity check) with a small transmission period crucial to achieve SDH/SONET-like fault detection speed? If yes, then the platform requires either a powerful processor or specific hardware assist to process in-service and out-of-service OAM protocol data units (PDU) continually for tasks such as frame loss measurement and delay/jitter measurement in a scalable fashion. MPLS-TP Continuity Check does require this.

Is accurate synchronization through packet-based synchronization transport mechanism like IEEE1588v2 important? If yes, the platform must make use of specific hardware to be able to insert and extract timestamp information with enough precision. Also a high quality oscillator needs to be used to minimize frequency optimize synchronization recovery.

With the advent of next generation silicon and processor technology, packet processing and forwarding (switching, bridging or routing) in the data plane is no longer a dominant cost factor when designing a new platform. When comparing platforms of dierent packet-based technologies, close attention must be paid to understand whether the platforms meet other non-data plane requirements, such as those listed above, which are equally pivotal when building a mission-critical network. When all factors are considered, the platform cost dierence should be negligible.

As well, since IP/MPLS supports layers 1 through 3 and MPLS-TP only supports layers 1 and 2, there is significant cost to having separate layer 1 and 2 from layer 3 solutions including equipment, management, training, sparing, etc. This is discussed further in the document.

Other values brought by IP/MPLS

As mentioned, MPLS-TP generally ignores the end to end requirements, mainly for IP applications and this is where IP/

57

Because of its native IP capability, IP/MPLS can oer a more comprehensive cybersecurity solution to mission critical networks.

Illustration 4.5 Cybersecurity enabled by IP/MPLS

MPLS fulfills the complete range of services for mission-critical applications.

IP Awareness:

Typically MPLS-TP has no IP awareness and therefore, for all applications that are IP based (eSCADA, CBTC, VoIP, meter reading, CCTV, PMUs...) customers using an MPLS-TP path will have to add an IP layer on top. Therefore, the network will look like the one in Illustration 4.7.

Such architecture creates issues because the two layers operate independently meaning that there is absolutely no dynamic

collaboration between the two layers. Main concerns are the following:

MPLS-TP provides poor eciency of bandwidth because there is a need for native IP trac to be transported between the two layers and typically the end to end path will likely not be optimized because it is fixed.

58

Whatever the application requirements at L2 or at L3, IP/MPLS supports all of them, either for point-to-point or point-to-multipoint applications.

Illustration 4.6 IP/MPLS supports all services at L2 and 3

Using separate technologies for aggregation and edge plus core leads to more complexity and is more dicult to manage.

Illustration 4.7 Overlay IP network on top of MPLS-TP Transport

End to end resiliency for IP applications will be triggered by MPLS-TP features and Layer 3 protocols, typically OSPF, therefore, convergence times can be much higher than 1 second in many scenarios which may create big outages for IP applications such as operational telephony over IP. This can create impacts on latency in the case where the lower layer backup scenarios are dierent than the upper layer scenarios.

Provisioning of services will be considerably more complex as there is a need to fully mesh the IP routers which will typically require a lot of LSP manual provisioning.

Troubleshooting will be more complex as there is a need to use dierent tools and dierent technologies.

Most private network operators need to segment their IP applications in dierent networks for organizational or regulatory issues including CCTV, intranet, telephony, financial payment applications, etc. With a basic IP network, this is not possible unless using complex VRF features which, again are very statically configured hop by hop and this activity doesnt scale well.

Most vendors of MPLS-TP have little IP knowledge and therefore railway, utility, government and other organizations will have to deal multiple vendors and their respective network management tools which can lead to a lack of clear ownership when a problem arises. IP/MPLS natively understands IP trac

through IP-VPNs and therefore can ensure through a single technology (single vendor and, potentially, single management tool) a fully optimized transport of data end to end. That is the typical solution that has been used by carrier networks for more than ten years and has recently become the solution of choice for utilities, government organizations, railways and other mission-critical networks.

Fast provisioning of services: mainly applications such as SCADA, Phasor Measurement Units, GOOSE... are not only point to point but becoming more point to multipoint and often require the flexibility for disaster recovery scenarios. Therefore, doing manual provisioning like MPLS-TP for such applications doesnt scale at all. That is why for these applications, having the choice to adapt provisioning from very manual to fully dynamic, as IP/MPLS provides, answers all needs of all applications. MPLS-TP can only provide solutions for some challenges of some applications but typically does not take a real end to end view of all requirements for railway, utilities and defense applications in a network.

Multiple failure scenarios: IP/MPLS supports many scenarios for resiliency and always ensures that as long as there is a path available between a source and a destination, the trac may use it, if it makes sense for the application. MPLS-TP has only one alternate path, if the first one fails, which may not be enough to reach 99.999% availability expected by mission-

59

critical and safety applications. It is important to note that in very meshed networks, ensuring that the primary LSP does not share any path or device with its redundant LSP is not always that easy to control. Relying on manual provisioning or on unproven network management tools can be very risky compared with the much proven and dynamic technology of IP/MPLS, with LDP for example, that has years of eective demonstration of resiliency capabilities.

Multicast Awareness: the number of video cameras is increasing heavily in many organizations for CCTV and is the most bandwidth hungry application that may be transported over a mission-critical communications network. In order to optimize any video trac delivery, IP has been enhanced with multicast protocols which have a simple role of ensuring that a video stream is transported once over a physical link independently of the number of receivers being registered for the stream. The multicast protocols basically use the dierent network equipment (routers, switches) to replicate the trac. While IP/MPLS has native multicast capabilities, MPLS-TP has none, which means that for example if a CCTV camera stream has to be stored in two locations (for resiliency) and has to be monitored by two OCCs, the trac of each camera will be sent four times over the links in access and core. Multiply this by 3Mbps, which is the average bandwidth for good quality video, you realize that with 100 cameras for example, you will generate 900 Mbps more trac with MPLS-TP than with IP/MPLS.

Clearly, a major impact on the cost of equipment and bandwidth, a fact often not mentioned by promoters of MPLS-TP technology.

Alcatel-Lucent added value to a standard IP/MPLS implementation

To make a clear and fair comparison between MPLS-TP and