Performance of a full-hardware PTP implementation 3 ......2 [5], by using IEC 62439-3 [2] as explained IEC 62439-3 consists of two redundanc zero switchover delay, PRP (Parallel Redu

Performance of a for an IEC 62439-

auProf. Dr. Hubert Kirrman

ABB Research Center, Baden [email protected]

Claudio Honegger ABB Research Center, Baden Sw

[email protected]

Abstract—Seamless redundancy and synchronization are integrated fully in hardwaprocessor support. The interaction betweenclock synchronization is explained. Redundamessages are not be discarded, but used accuracy. Measurement results confirm thconcept. Performance of the full-hardware compared with that of a conventional implementation.

Keywords: IEEE 1588; IEC 61588; IEC 61PRP; HSR; clock redundancy; FPGA; power prof

I. INTRODUCTION Substation automation networks require

and precise clock synchronization. Until nowand redundancy were treated separately.

IEC 61850 [1], the communication standSubstation Automation, will specify 1 distribution over an IEEE 1588 profile [3] in

Since IEC 61850-8-1 (GOOSE) and IEtransmit time-critical data over Ethernet IEEE(physical and link layer), they also specify re2 [5], by using IEC 62439-3 [2] as explained

IEC 62439-3 consists of two redundanczero switchover delay, PRP (Parallel Reduand HSR (High-availability Seamless Roperate on the same principle of parallel tranindependent paths. This scheme works wellbut presents a challenge for clock synchronidoes not consider redundant PTP messages [7

IEC 62439-3 Annex A defines how to PTP messages when using IEEE 1588 psubstation automation.

This paper describes a first implementatioAnnex A and shows that this concept holds it

This paper shows the benefits oHSR/PRP/PTP transparent and ordinary clock

This paper show that the whole stack canFPGA with benefits in terms of resources and

full-hardware PTP imp-3 redundant IEC 6185utomation network

nn witzerland om

witzerland com

DiaABB Research Cent

diana.ilie@Ioannis S

ABB Research Centioannis.sotiropo

precise clock are (FPGA) with no n redundancy and nt synchronization to improve clock

he validity of the implementation is software-hardware

1850; IEC 62439-3; file;

e high availability w, synchronization

dard for Electrical µs precise time its next edition. C 61850-9-2 (SV) E 802.3 [4] layer 2 dundancy on layer in [6]. cy protocols with undancy Protocol)

Redundancy), that nsmission over two l for usual traffic, ization since 1588 7], [8].

handle redundant profile suited for

on of IEC 62439-3 ts promises. of an integrated k in FPGA. n be handled in the d performance.

II. NETWORK RED

A. Parallel Redundancy ProtoFigure 1 shows a typic

Redundancy-enabled nodes, cnodes with PRP) have two portLANs, LAN_A and LAN_B.

Both LANs are switched Etof nodes and bridges operatexecutes the clock synchronizAnnex J.4 (default peer-to-peerdevices within the LANs are un

Figure 1 – Parallel redundancy Prot

The source of a frame send“A”-frame and “B”-frame. Thsame time and travel indepenaccepts the first frame and discIn case of loss, the application with the remaining frame undis

To identify duplicates, eanumber in a redundancy controSince a normal application igno

plementation 50 substation

ana Ilie ter, Baden, Switzerland

@ch.abb.com Sotiropoulos ter, Baden, Switzerland [email protected]

DUNDANCY PRINCIPLES

ocol (PRP) Principle cal duplicated PRP network. called DANP (doubly attached ts connected to two independent

thernet with an arbitrary number ting with RSTP. Each bridge zation according to IEEE 1588 r profile, one or two steps). The naware of the PRP protocol.

tocol (PRP) principle (IEC 62439-3).

ds a copy of it over each port, an he frames are sent at about the ndently. The DANP destination cards the duplicate (if it arrives). on the destination node operates

sturbed. ach frame carries a sequence

ol trailer appended to the frames. ores this trailer, it is unaware of

PRP. This allows to connect Singly Attachsuch as printers or laptops, to one LAN witho

Singly attached devices that need redundthrough RedBoxes (redundancy boxes) that the frames for them. RedBoxes allow to attaor complete LAN segments and also interbelow).

B. PRP node structure

Figure 2. Node structure of PRP (IEC 62

Each DANP (Figure 2) has two Ethernetsame MAC address and present the samtherefore, PRP is a layer 2 redundancy.protocols above layer 2 to operate as iredundancy and simplifies engineering.

An additional sub-layer is introduced unmodified) link layer, the LRE (Link Rethat handles both Ethernet controllers and interface towards the upper layers as ainterface.

In a source node, the LRE duplicates thefrom the layers above and sends the frames oat nearly the same time.

In a destination node, the LRE receivefrom both adapters within some time skew sthe networks are different. It keeps track through the duplicate tables, that are typicallytables. The LRE discards the second frame ooffload the application (it would work discard but not as well).

If a link or a port is damaged, the LREframes over the other path, data keep flowinpath without influence on the application, tnote that the duplicate did not arrive for diagn

The LRE for PRP could be implementedwithin the driver or the communication pintroduces a negligible delay.

hed Nodes (SANs) out modification. dancy are attached duplicate / merge

ach single devices rface to HSR (see

2439-3)

t adapters with the me IP address(es);

This allows all if there were no

in the (otherwise edundancy Entity), presents the same

a single Ethernet

e frame it received over both adapters

es the same frame since the delays in of the duplicates

y arranged as hash of a pair, mainly to without duplicate

E will still receive ng over the healthy the LRE can take nostics purpose. d in software, e.g.

processor, since it

C. High-availability Seamless HSR (IEC 62439-3, C

redundancy with only one addiduplication of frames and disca

Figure 3. Example of H

Each redundancy-enabled attached node with HSR) hascapabilities. The nodes are daistopologies also exist. Each DAcapability described in IECIEEE 802.1D [9].

The sender of a frame insequeue of each port. The two fratime and travel in opposite direc

In the fault-free state of treceives two identical frames, pits application and discards the

A DANH should not forwfrom the ring, when: - the node is the sole destination- the node itself injected the fram- the node already forwarded th

Every node is responsible tthem from the ring. Thus, comof any frame, any link or any destination.

The network delay is the prdelays (max. 5 µs) by the numb

Although the average delaythat of a simple ring, the worstlink or node is given by the tota

To keep this delay at a rtransmission of sampled valueIEC 62439-3 prescribes that eaframes within 5 µs when there that nodes provide a cut-throprovided by hardware support.

Cut-through reduces the avdoes not improve the worst happens when all nodes at the

Redundancy (HSR) principle Clause 5) provides seamless itional link. As PRP, it relies on

arding of duplicates (Figure 3).

HSR ring (IEC 62439-3).

node, called a DANH (doubly s at least two ports of similar sy-chained into a ring, but other

ANH implements the forwarding C 62439-3 [2] according to

rts a copy of it into the sending ames are sent at nearly the same ction. the ring, each destination node passes the first frame of a pair to duplicate.

ward a frame which is received

n of the frame me into the ring

he frame. to detect duplicates and remove

mmunication can sustain the loss node that is neither source nor

roduct of the individual bridging ber of transit nodes. y in a healthy HSR ring is half t-case delay in case of loss of a

al number of nodes in the ring. rate acceptable for time-critical es (4.8 kHz, or 208 µs interval), ch node in the ring forwards the is no other traffic. This requires ugh bridging that can only be verage forwarding delay, but it case propagation delay, which e same time inject a maximum

size frame, and delay the forwarded framesring traffic has priority.

D. HSR Node structure As Figure 4 shows, the structure of an H

to that of a PRP node (Figure 2).

Figure 4. Node structure of HSR (IEC 62

In HSR, there is addition to PRP a bforwards frames from port A to port BIEC 61439-3 prescribes a forwarding delay when there is no other traffic, to keep propagation delays along the ring low. Srequires a cut-through mode, i.e. a frame is fis entirely received.

Therefore, if an Ethernet protocol for communication system utilizing HSR wants cut-through properties, it has to constrain ttheir frames in a pre-allocated time windinstance using a common precision clock (see

The duplicate detection is more elaboratePRP since a malfunction of the duplicate detring flooding.

Figure 4 shows the duplicate detectcomponent. There may also exist a duplicaport, a solution that scales better with the num

III. IEEE 1588 CLOCKS SYNCHRO

Clock synchronization in PRP and HSR1588 V2, using the profile of Annex J.4 [3], [

For PRP and HSR, IEC 62439-3 Annexprofile suitable for time-critical systems innetwork devices between GrandMaster and OTransparent Clocks, using:

• Ethernet layer 2 communication (no U• Multicast PTP messages (no unicast)• Peer-to-peer path delay (no end-to-en• One-step or two-step both allowed • Best Master Clock with default settin

s, even though the

SR node is similar

2439-3)

ridging logic that B and vice-versa.

of less than 5 µs the sum of the

Such a low delay forwarded before it

a hard real-time to exploit fully its the nodes to send

dow (TDMA), for e [2]). ed in HSR than in tection can cause a

tor as a central ate table for each

mber of ports.

ONIZATION R relies on IEEE [10]. x A defines a PTP n substations. All Ordinary Clock are

UDP)

nd delay)

ngs

IEEE 1588 assumes thaFollow_Up take the same occasionally defeated in RSTPlink or bridge failure, but in PRcase but this is not foreseen in I

The problems stated in IEC 62439-3 Annex A, as follo

In PRP, the GrandMasterconnected to LAN_A and LAN

The PRP duplicate discardmessages because the bridges Thus, they do not preserve the PSync messages and append no t

Figure 5. Cl

The three frames (Announbelong to the same LAN, otherwbe consistent. The Sync meswhether they come over LAN_A

Although Announce messaLANs, some derived standardand Announce could be differen

Pdelay_Req and Pdelay_therefore LAN-specific. A Ddelays on LAN_A and LAN_Btwo-step Sync and Pdelay tomessages.

Therefore, the ordinary clmessages from each port separclocks, and listens to only one o

When the DANP detects thsame GrandMaster, it uses bothand improve the clock accuracy

Indeed, although Sync A ansource, using both improves

at the Announce, Sync and path. This condition can be

P during reconfiguration after a RP and HSR, this is the normal IEEE 1588.

[7] have been addressed in ows. r is assumed to be a DANP

N_B, as Figure 5 shows. d method cannot apply to PTP in the LAN are not PRP-aware. PRP trailer when forwarding the trailer to their Follow_Up.

locks in PRP

ce, Sync and Follow_Up) must wise the correction field will not ssages have different contents A or LAN_B.

ages should be identical on both ds allow dynamic modification nt on LAN_A and LAN_B. _Resp are link-specific and

DANP sees two different link B. To simplify logic, it converts o one-step if it receives such

lock in the DANP treats PTP rately, as coming from different of them, or accepts both. at PTP messages come from the h Sync messages to reduce jitter y. nd Sync B come from the same accuracy since the two Syncs

suffered different stochastic network delays clock inaccuracies or path variations.

In hardware measurements the network jdistribution, which is used to approximate the

Figure 6 shows the effect of losing one Lsynchronization when a large number of thave been crossed in each LAN.

In a healthy PRP/HSR network, the jitter statistical effect. For a known distribution ofthe more values are considered, the better the

Figure 6. Clock accuracy in PRP (norma

It could also be that the clocks are differeLAN_B. The loss of the link to the grandmcauses the election of a backup master on case, the DANP considers only the best quaan extension to IEEE 1588, the DANPs alsoMaster Clock algorithm to select among diffe

A. HSR and clocks In HSR, the time distribution is similar

messages injected into the ring by the Granboth directions (Figure 7). They have the sasequence number. Each DANH has both a Tand an Ordinary Clock for itself. This is calle

While the preferred transmission mode step are also allowed. A one-step capable DAstep messages to one-step.

As in PRP, each DANH treats the PTP mdirection separately. The Ordinary Clock uimprove accuracy when both come from thsituation in which two different mastersmessages is handled as in PRP.

In principle, PTP messages need no HSbeen maintained for removing truncated mess

Compared to a non-redundant clock, an handle double as many PTP messages and exThis is especially felt when the GrandMaster masters cause a “PTP-storm” of messages thhave difficulty to cope with.

due to PHY jitter,

jitter has a normal e real value. LAN on the clock transparent clocks

is smaller due to a f random variable,

e approximation.

alized)

ent in LAN_A and master in one LAN

that LAN. In that ality master. So, as o execute the Best erent masters.

to PRP. The PTP ndMaster travel in ame HSR and PTP Transparent Clock

ed a Hybrid Clock. is one-step, two-

ANH converts two-

messages for each uses both Syncs to he same clock. A s send Announce

SR tag, but it has sages. HSR node has to

xecute the BMCA. fails. The back-up at most processors

Figure 7. Cl

B. PRP to HSR connection. A ring can be connected t

Figure 8 shows. Two RedBoxpoint of failure. Since each Redring, four copy of a frame woframe reaches a port that has discarded, so that the overall trframes. On reception, a RedBox

The PRP trailer, resp. the Horiginating HSR ring or PRP ne

This information is lost forHSR nodes must handle four GrandMaster they come fromcould perform an extended sequence number of the PTP here also, using all four Syncs r

Figure 8. Clocks in P

end node

TC

end node

OC

TC

end node

redundant master clock

„A“-frame„B“-frame

GPS

MC

TC

locks in HSR

o a duplicated PRP network as es are needed to avoid a single dBox injects two frames into the ould circulate, but as soon as a already sent a copy, it will be

raffic consists just of the pair of x only forwards one copy.

HSR tag carry an identifier of the etwork to prevent reinjection. r the PTP messages. Therefore,

Syncs and detect from which m. An HSR RedBox therefore

duplicate discard using the messages to reduce traffic, but

reduces jitter.

PRP and HSR coupling

e

OC

end node

OC

TC

ks

switch

interlink

end node

TC TC

MC

IV. NODE IMPLEMENTATIO

The close coupling between bridging andintegrated implementation of HSR and PAlthough PRP can be realized in software onand PTP logic is in place, adding the PRP log

Figure 9 shows a combined HSR-PTPports. Three transparent clocks compute thbetween each port pair. Each port keeps trackThe ordinary clock is syntonized and keepscan serve as a real-time clock for the applicagenerate the sampling pulses for the ADC and

The choice of FPGA and processor is a cthe available hardware. The FPGA is an Awith a 66 MHz clock, 55’000 LE and 256 xMicrel KSZ8001 PHY operate at 100 Mbit/clock. Time-stamping is done in the FPGA.

The challenges were to reduce the elements, cope with different clock domains own clock), and provide a future-proof arcaccommodate higher speeds and a larger num

The inaccuracy introduced by the trancurrently below 50 ns per node and could PHYs that recognize the time-stamping point

Figure 9. Node layout (one directio

A. Hybrid PTP Implementation A hybrid software-hardware implementat

the-art implementation for a system that usand PTP ([8]). In such a system, a part of timplemented in logic, and the rest is implemeblock diagram of the hybrid implementatioFigure 10.

The node has 3 external ports, 2 of them and one as a non-redundant port. This port isCPU that executes the application, in parautomation protection. The HSR/PRP core discards duplicates for each port is implem

ON d clock calls for an PTP in hardware. nly, once the HSR

gic is trivial. P node with three he residence delay k of the peer delay. s the exact time, it ation processor and d the 1PPS output.

convenience due to Altera Cyclone III x 9K of RAM. The /s with a 25 MHz

amount of logic (each PHY has its

chitecture that can mber of ports. nsparent clocks is be improved with

t in the messages.

on)

tion is the state-of-es both HSR/PRP

the functionality is ented by a CPU. A on is presented in

used as HSR ports s connected with a rticular substation that identifies and ented in hardware

logic. To accept indifferently oconnected to a 2-to-1 step convstep Sync frames are handledconnected with a block of Ordinary Clock that is synchronis implemented in hardwaresoftware are the management sAlgorithm. The CPU receivesdecodes them and executes the

In case the current node isgenerates the Sync and Annouthe HSR/PRP core, which tagsduplicate discard tables and thother ports, so this node’s clock

The CPU can be an externathe FPGA, or a softcore CPU the FPGA. Both approaches hfirst approach, the same CPUlayer also executes the BMC alg

Figure 10 - Hybrid PTP impRetaining part of PTP funct

since it can be easily implemeCPU can handles layer 2 megeneration of PTP traffic can ea

B. Full hardware PTP implemA hardware-only PTP switc

was also implemented. The blosame as in Figure 10, only thBMC-/Management logic direc

C. Comparison of the hybrid aimplementation The hybrid approach prov

update of the BMC algorithm,

ne and two-step clocks, a port is version block. Therefore, only 1-d internally. Each port is also Transparent Clock. The PTP

nized by the PTP Sync messages e. The parts implemented in stack and the Best Master Clock s Announce and Sync frames, BMC algorithm.

s selected as a master, the CPU unce frames, which pass through s them and registers them in the hen transmits them through the k is distributed over the network. al hardcore CPU, connected with

(NIOS) instantiated directly in have been implemented. In the U that executes the application gorithm.

plementation with redundancy tionality in a CPU is convenient, nted if the network stack of the

essages. However, reception or asily overload the CPU.

mentation ch with redundancy (HSR/PRP) ock diagram of this design is the at the NIOS is replaced by the

ctly implemented in VHDL.

and the hardware-only

vides an easy maintenance and as only the software code has to

be changed. On the other hand, the layer 2 handling of the netstack requires a lot of effort and intensive testing.

The hardware-only approach relieves the processor from the PTP related tasks, which are significant. It provides also better area and memory utilization, faster system response and less programming effort. The softcore processor can be removed, leaving place for more RAM blocks and logical elements. System response is better since the hardware BMCA reacts faster. Table 2 shows the resource utilization of a hybrid solution, when a softcore CPU (Altera’s NIOS II) is used. Saving the resources of the CPU and its communication with the FPGA core is an important optimization, as shown on Table 3

Another advantage of the hardware-only implementation is portability. While adding PTP to a CPU stack requires many changes and additions to the existing stack, integrating the PTP IP requires only the configuration and interface signal connections on the FPGA.

HSR nodes and transparent clocks both forward the ingressing traffic from one port to another. When a node supports both modes, the two modes can be treated separately/independently. In this case, the forwarding delay (measured between the time-stamping point of the ingressing and egressing frame) is the sum of each mode’s delay (which is caused by the frame buffering, until required information is decoded from it or controllers’ responses about forwarding or dropping the frame) . The residence delays can be reduced by unifying operations that buffer the ingressing frames. (Table 1).

TABLE 1 - AVERAGE FORWARDING DELAY PER NODE (FULL HW)

Components Forwarding Delay HSR ~ 4 µs Transparent Clock ~ 2.5 µs HSR+Transparent Clock (Independent Functions) ~ 6.5 µs HSR+Transparent Clock (Unified Functions) ~ 4.2 µs

TABLE 2 – RESOURCE UTILIZATION OF A HYBRID HW-SW SOLUTION

Components Logic Elements

Block RAMs (M9K)

HSR/PRP (2 Red Ports – 2 Interlinks) 24,000 70 PTP Transparent Clock (3 Ports) 16,000 21 PTP 2-1 Step Converter (3 Ports) 3,700 6 PTP Ordinary Clock 4,000 2 PTP BMC + PTP Frame Generator on CPU 3,000 141 Communication of CPU with PTP on FPGA 3,000 12

TABLE 3 – RESOURCE UTILIZATION OF FULL HW SOLUTION

Components Logic Elements

Block RAMs (M9K)

HSR/PRP (2 Red Ports – 2 Interlinks) 24,000 70 PTP Transparent Clock (3 Ports) 16,000 21 PTP 2-1 Step Converter (3 Ports) 3,700 6 PTP Ordinary Clock 4,000 2 PTP BMC + PTP Frame Generator 3,000 3

D. Scalability This concept allows to build devices that connect to

multiple HSR rings, replicating the logic for each port, with practically no overhead.

V. CONCLUSIONS A full-hardware implementation of a combined redundancy

IEC 62439-3 with IEEE 1588 clock synchronization is cost effective and can be used for new devices and retrofit. It provides synchronization and redundancy with no additional burden on the application processors.

The modular design allows to build HSR nodes with a large number of ports, as well as RedBoxes.

As special advantages, this design is technology and operating system independent, has a good form factor and eases maintenance while providing better stability.

Future development would benefit from PHY that provide a synchronization pulse according to IEEE 1588. A 1 Gbit/s design is no real challenge, it has only a cost impact since faster logic and larger memories are needed. References

[1] IEC 61850 Ed.2: (Part 5, 8-1 and 9-2) Communication networks and systems for power utility automation, International Electrotechnical Commission, Geneva, 2012

[2] IEC 62439-3 “High Availability Automation Networks – PRP & HSR”, International Electrotechnical Commission, Geneva, 2012

[3] IEC 61588, Precision Time Protocol (PTP), version 2, International Electrotechnical Commission, Geneva, 2008.

[4] The Institute of Electrical and Electronic Engineers, “CSMA/CD access method and physical layer specifications. IEEE Std 802.3”, 2005.

[5] H. Kirrmann, C. Hoga, O. Kleineberg; H. Weibel “HSR: Industrial Ethernet redundancy with zero recovery time and low cost (High availability Seamless Ring, IEC 62439-3)”, ETFA 2009, 2009

[6] Kirrmann, H.; Dzung, D.; “Selecting a Standard Redundancy Method for Highly Available Industrial Networks”, 2006 IEEE International Workshop on Factory Communication Systems, June 27, 2006 Page(s):386 – 390

[7] Abdul Amin, “Integration of HSR and IEEE 1588 over Ethernet networks”, ISPCS 2010.

[8] Hans Weibel, Sven Meier, “IEEE 1588 applied in the environment of high availability LANs, International IEEE Symposium on Precision Clock Synchronization for Measurement, October 2007.

[9] The Institute of Electrical and Electronic Engineers, “ ANSI/IEEE Std 801.2D, Media Access Control (MAC) Bridges”, 2004.

[10] Tournier J.C, Weber K., Hoga C. “Precise Time Synchronization on a High Available Redundant Ring Protocol”, 2009 IEEE International Symposium on Precise Clock Synchronization. Brescia, Italy, September 2009.