Ali 2008

Enhancement of Industrial Ethernet Performance UsingMulticasting/VLAN techniques

Qutaiba I. AliComputer Eng. Dept.Mosul University.

Mosul, IraqQQQ1 @maktoob.com

Abstract- This paper studies Industrial Ethernet performanceunder different circumstances using the network simulationpackage(OPNET).It was assumed that industrial Ethernet isused to connect the different components of an electricalsubstation automation system. In the first simulation case,unicast transmission procedure is assumed. Unicast consumesCPU power and resources of the industrial node. In addition, ifthe packet production rate is high, queuing delay is created atdifferent network layers of the industrial node. All this addsextra delay to the total value of the latency. Then, acomprehensive study to the effect of using multicasting wasmade. MulticastingNVLAN techniques are used to enhance thereal time performance of industrial Ethernet by modifyingpacket generation process. Industrial nodes use multicastingtechnique to generate a packet forwarded to all members of amulticast group in the same time. Meanwhile, VLAN techniqueis used to forward the multicast traffic to its intendeddestinations only.

Keywords Industrial Ethernet, Latency, Multicasting,Virtual LAN

I. INTRODUCTIONMany field bus vendors are moving forward to unify

their efforts to establish a common network for theirindustrial solutions. Industrial Ethernet was chosen to bethat solution. Industrial Ethernet uses all types of theprotocols of traditional Ethernet including the TransportControl Protocol (TCP), the Internet Protocol (IP) and themedia access and signaling technologies found in allEthernet networks [ 1 ].

Much attention has been focused on the use of Ethernettechnology directly at the device level. The decreasing costand increasing capabilities of network interfaces andmicroprocessors have accelerated the movement ofcommunications network connections down to theinstrument and device level. The Ethernet-Based IntelligentElectronic Device (IED) is a measurement or I/0 devicewith an Ethernet connection directly on the device itself.This approach provides a relatively inexpensive option fornetworked data acquisition, and provides greater versatilityin terms of locating the measurement device in size-constrained areas or harsh environments [2].

11. TCP/IP AND UDP/IP USAGE IN INDUSTRIAL ETHERNETA de facto protocol standard for network

communication is the IETF protocol suite, usually calledTransmission Control Protocol/Internet Protocol (TCP/IP).

Basil Sh. MahmoodElectronic Eng. College

Mosul University.Mosul, Iraq

Basil Mahmoodgyahoo.com

This suite contains two transport protocols: TransmissionControl Protocol (TCP) and User datagram protocol(UDP). The main differences are that TCP is slow, reliable,and connection-oriented whereas UDP is fast, unreliable,and connectionless. In the case of high-speed measurementdata, TCP is less useful due to the protocol overheadinvolved. Therefore, UDP was chosen as the measurementdata real-time protocol. This turns out to be more thansatisfactory, since measurement values is sampled at a veryhigh rate. Therefore, if a data set is lost, another set will becoming along shortly [3, 4]

III. INTRODUCTION TO SUBSTATION AUTOMATION SYSTEMA substation is a large number of switchgears

controlled, supervised and protected by a SubstationAutomation System (SA). The Substation AutomationSystem (SA) is a network of functions realized insidedevices strongly interacting as a system. The SubstationAutomation System (SA) comprises full station and bayprotection as well as control, monitoring andcommunication functions and provides all functionsrequired for the safe and reliable operation of thesubstations. A typical high voltage substation connects 3-10transmission lines (feeder bay); two or more powertransformers (transformer bay) and has a physical dimensionof hundreds of meters [5].

For protection purposes, reaction times of the completesystem must be in the order of 4-10 msec. (1 msec. for extrahigh performance systems) between fault occurrence andcircuit breaker deenergising. Other monitoring includes thestate of the electrical process, currents and voltages, statusinformation from circuit breakers, gas insulation units andmore[6].

Communication requirements for the substation wereimplemented using different techniques, such as point-to-point and field bus systems (especially Modbus type[7]).Industrial Ethernet was entered strongly into this field andunified efforts of different organizations result thedevelopment of the International Standard - IEC 61850 -

Communication Networks and Systems in Substations[6].A typical layout for a substation consists of several sub

networks. Each sub network (bay) consists of the followingcomponents[5,7,8]:1.(8) IEDs used as sensors (Sn) to measure

different quantities at different sampling rates.

2.(3) IEDs work as actuators (ACn) for differentpurposes (e.g., circuit breaker).

3.One local controller which has the followingfunctions:

* Connection to remote centers (SCADA system purposes)* Automation and local control

Also, there is a global controller responsible for thewhole substation's automation and protection functions. Inaddition, global controller can take the tripping actions ofany local controller in the case of its malfunction.In the control room, there is a computer used for:* Operator interface for monitoring and control of the entiresubstation* Maintenance interface* System administration interface.This is called Human Machine Interface (HMI).

IV. SIMULATING INDUSTRIAL ETHERNETPERFORMANCE IN SUBSTATION AUTOMATIONThe research tool used in this paper is OPNET

(OPtimized NETworks). It is an advanced package thatallows the user to design and study communicationnetworks, devices, protocols, and application [4].

An OPNET model (represents five sub networkssubstation automation system) is built to test theperformance of Industrial Ethernet under hard real timeconditions (with lmSec. as the dead time). The differentparts of the model have the performance listed in table (1)[7].

TABLE I. INDUSTRIAL ETHERNET PERFORMANCE

EthernetType ~~Fast Ethernet (Data Rate:1I00Ethernet Type Mbps)

Sensors packet processing rate 5000 Packet/Sec.

Actuators packet processing rate 5000 Packet/Sec.

Local controller's packet processing 10000 Packet/Sec.rateGlobal controller's packet 30000 Packet/Sec.processing rate

UTP for short distances-Fiberoptics for long distances

The traffic pattern (which represents the controlactivities of the different parts of the substation automationsystem) presented in references [7, 9, 10] is adopted and itcan be summarized as listed in table (2)(The detaileddescription of the protection algorithms for the substation isbeyond the scope of this paper).

TABLE II TRAFFIC PATTERN OF A TYPICAL SUBSTATIONAUTOMATION SYSTEM

Source Destination Packet / PacketSec. Length

S 1-S2-S3 Local Cont.- AC1-AC2- 1000 32 ByteGlobal Cont.

All Sensors Local Cont.-AC1-AC2 10 32 Byte

Local Cont. AC3 250 16 ByteGlobal Cont. AC3's in all subnet's 250 16 Byte

HMI S1-S2-S3-Local Cont.- 1 file each 1MbyteGlobal Cont.

Local Cont.- HMI 2 file/min. 1MbyteGlobal Cont.

As listed in the table, UDP protocol is used to transfertime critical data (measured samples) while TCP is used totransfer administrator's configuration information to somenodes. SCADA information is transferred periodically formglobal controller and from each local controller (summeryof Bay data) to HMI in the control room for monitoring andcontrol purposes. The traffic in the table, represent thetraffic in one bay (sub network) and it is repeated in theother bays.

The metrics used to evaluate the performance ofIndustrial Ethernet are [7]:*Local Latency: {the latency measured from theapplication layer in the sensors to the application layer inthe local controller} + {the latency measured from theapplication layer in the local controller to the applicationlayer in the circuit breaker}.*Global Latency: {the latency measured from theapplication layer in the sensors to the application layer inthe global controller} + {the latency measured from theapplication layer in the global controller to the applicationlayer in the circuit breaker}.

V. COMMENTS ON THE SIMULATION RESULTSFrom running the simulation model, different statistics

(which explain the behavior of the system) were collected asfollow:A. Traffic map ofthe Substation Automation system

The complete traffic flow of the control data packets onthe network is shown in figure (1). The values shown in thefigure are calculated by the simulation program as follows:By taking node (S1) as an example, the throughput (actualdata rate on a network channel) can be calculated bymultiplying the packet production rate (4030 Packet/Sec.)by the packet size (32 Byte + headers at different network'sstack layers), which results in (3.8 Mbps).

The throughput on each segment represents lowutilization of the whole channel capacity (100 Mbps).Thatmeans, congestion problem (nodes traffic exceed channelcapacity) is avoided and hence, Ethernet delay has aminimal effect on the total latency. FTP traffic causes atransit increase in the throughput values shown in the trafficmap.

S2 ps

S5 0~02

0-0

OSE 2 N -

O ftp -*

0------ L -1 SUBIFigure 1. Traffic Map ofthe Reference Model

B. Local and Global Latency:Figure (2) shows the change in local and global

latencies over simulation time. The following observationscould be extracted from the graphs:

Flp 7j..f.

t:m

*Most of the latency is concentrated inside the node'snetwork stack.*Latency graphs have two regions: constant latency regionand unstable region (since the system fails to respond withinthe dead line time of (lmSec.)).

Constant latency region represents the system behaviorwhen the traffic on the network is of the real time data only.The value of this latency is less than (lmSec.) whichresponds successfully to the real time bounds of the system.However, this value is still relatively high, because for eachmeasured sample , the sensors IEDs generate four packetsaddressed to four different destinations( refer to table(2)).This gives a rise to the node's CPU utilization (seeTable(3)) and generates more queuing delay inside thenodes, and all that increase latency.

The unstable behavior of the system shown inFigure(2), is coming up from the highly loaded nodes thatare subjected to more traffic during the file transferoperation which adds more delay to the packet generationoperation and hence, increased latency. File transferprotocol (FTP) is used to handle the operation of filetransfer and it needs a maximum time of (3.5 Sec. ) tocomplete its task.*Global latency has a lower value than local latency. This iscaused by the higher processing capabilities of the globalcontroller compared to local controller. On the other hand,the Ethernet delay to the global controller has an averagevalue of (67 [tSec.) while the average value of Ethernetdelay on the local controller is equal to (46 [tSec.) .Thisassures that Ethernet delay (in this model) has a minor effecton the total latency.*The small peaks in the graphs were caused by the uploadoperation (SCADA information transfer) from thecontrollers to the control room.*The Sensor nodes (especially S1-S3) suffer from a higherutilization of their associated CPU more than other nodes,See table (3)

1.8 (~~~~FTP Downioe1 .6

! 1 .41.2 i FTP Um10.80.60.40.20

0 50 100 150 200

Simulation Time(Sec.)

1~Global Latency - 'Local LatencoFigure 2. Latancy Variation of the Reference Model

TABLE III. CPU UTILIZATION OF THE SUBSTATION NODESNode Associated CPU Utilization %lGlobal Controller 60%0Local Controller 34%0S1-S3 83%lS4-S8 1%tAC1,AC2 31%AC3 10%

VI. MODIFYING INDUSTRIAL ETHERNET BEHAVIORUSING MULTICASTING / VLAN TECHNIQUES

It is noted in figure (2) that latency values are stillrelatively high. Most of the remaining latency comes fromthe originating nodes (sensors). The packet generationtechnique used by the sensors follows the unicast procedureas follows [f1]:1. Measured samples are converted to data bits in the

sensor's application layer and some processing tasksare implemented on them (according to the specificapplication).

2. At transport layer, multiple segments is createddepending on the number of the intended destinations.For each segment, a transport layer header is addedwith all necessary calculations.

3. Each segment is forwarded to the network layer. IPAddresses are added in addition to all other networklayer's header parts. At this point, segments are calleddatagrams.

4. Datagrams now reach the data link layer, which addsmore headers (including M\AC addresses) and thensend them out ( as packets) to the transmissionmedium.

5. This sequence is true and repeated for all other copies,but with different destination addresses.Unicast procedure consumes CPU power and resources

of the industrial node (return to Table 3). In addition, if thepacket production rate is high, queuing delay is created atdifferent network layers of the industrial node. All this addsextra delay to the total value of the latency. Unicast isusually used when the node intends to send to a specificdestination If the node intends to send the same packet tomultiple receivers at the same time, multicasting techniqueis used. Today, this technique is used by various internetapplications, such as: video conferencing, corporatecommunications, distance learning, and distribution ofsoftware, and news. The main contribution of multicastingin these fields is to minimize channel utilization, and hence,improves internet services characteristics [12].

Some researchers [7, 13] suggested (as a future work)the use of multicasting to enhance the performance of thereal time systems. However, none of them studied the effectof such a suggestion. In this paper, a comprehensive studyto the effect of using multicasting was made.

A. Using Multicast in Industrial Ethernet

Multicast is based on the concept of grouping. Amulticast group is an arbitrary group of receivers thatexpresses an interest in receiving a particular data stream.This group has no physical or geographical boundaries thehosts can be located anywhere on the Internet or any privatenetwork [12].

IP multicast addresses specify a "set" of IP hosts thathave joined a group and are interested in receiving multicasttraffic designated for that particular group [12].

The Internet Assigned Numbers Authority (IANA)controls the assignment of IP multicast addresses. IANA hasassigned the IPv4 Class D address space to be used for IPmulticast. Therefore, all IP multicast group addresses fall inthe range from 224.0.0.0 through 239.255.255.255[12].Inthis paper, IP multicast was used to enhance the real timeperformance of Industrial Ethernet by modifying packetgeneration process. Nodes were arranged into several

multicast groups according to traffic pattern listed earlier intable (2). Table (4) detailed this arrangement. Accordinglywhen sensors generates a packet, its address is the value ofIP multicast group and therefore this packet is forwarded toall members of the multicast group in the same time.

TABLE IV. MULTICAST GROUPS FOR DIFFERENT NODESMulticast S1 - Local Controller - AC I - AC2 - Global ControllerGroup 1Multicast S2 - Local Controller - AC1 - AC2 - Global ControllerGroup 2Multicast S3-Local Controller - AC1 - AC2 - Global ControllerGroup 3Multicast S1 - Local Controller - AC I - AC2Group 4Multicast S2 - Local Controller - AC t - AC2Group 5Multicast S3- Local Controller - ACt - AC2Group 6Multicast S4 - Local Controller - ACt - AC2Group 7Multicast S5 - Local Controller - AC t - AC2Group 8Multicast S6 - Local Controller -AC t - AC2Group 9Multicast S7 - Local Controller -ACt - AC2Group 10Multicast S8 - Local Controller - ACt - AC2Group It

The multicast groups were fed into OPNET simulationenvironment. The results obtained from running thesimulation show that CPU utilization of the sensors falls(from 83% to 20%) which decrease the delay inside thosenodes. The positive effect of multicasting technique onlatency is shown in shown in figure (3).

09h Iq

0 7

QJ 0.64)

E 05-

C 04

:% 03

0.2-

0 1

-

0 50 100 150Simulation Time (Sec.)

I-Global Latency - Local Latency

Figure 3. Latency Variation in Multicasting case

However, Multicasting causes an increase in networktraffic. The reason behind this increase belongs to the factthat Ethernet switch do not have the ability to forwardmulticast traffic to its intended destinations only. Ethernetswitch keeps a list of the physical addresses of the nodesconnected to it, if it receives any packet carried a differentdestination address (including a multicast group address), itwill forward it to all other ports except the one which come

from. This is called "flooding" [13]. The effect of floodingon the network traffic can be clearly shown in the figure (4).

In flooding, the traffic on the network is increased andthe nodes receive more traffic, that most of it does notbelong to those nodes .This causes additional load on thenode's NIC, and hence, more delay. In addition, Ethernetdelay increased (especially queuing delay in the switches)

because more packets are competing on the same outputport. In the internet, the problem of flooding was solved byusing Internet Group Management Protocol Snooping(IGMPS) which is installed on the network's routers [14]. Inthe present system, routers were not used. The alternativesolution is to use virtual LAN (VLAN) technique.

DW 1*

SUJBI

Figure 4. Traffic Map in Multicasting Case

B. Virtual LAN OverviewVirtual LAN is a new technique based on the

segmentation of any switched LAN into multiple logicalLANs. Nodes that share some features or belong to a certaingroup can be assigned to the same VLAN. so that, they canreceive and transmit packets to the members of that VLANonly. In other words, the single broadcast domain in thetraditional switched LANs is segmented into multiplebroadcast domains. VLAN technique provides enhancednetwork features such as simplification of networkmanagement, controlled traffic activity and workgroup andnetwork security[ 15].

In general, there are three basic models for determininghow a packet gets assigned to a VLAN[15]:* Port-based VLANst MnAC address-based VLANsbProtocol-based VLANs (Layer 3 VLANs)

The VLAN membership of a packet is indicated by atag that is added to the packet. This method is defined in theIEEE standard 802. 1Q, approved in 1998. When a packetarrives at its local switch, the VLAN membership can bedetermined as port based, M\AC address-based or protocol-based. When the packet is transferred to other switches theVLAN membership can be defined through the tag that wasadded by the first switch [15].

C. Using Multicasting / VLAN in Industrial Ethernet

One of the benefits behind using VLAN technique isthe isolation of the traffic of different multicast groups. Thismay solve the flooding problem mentioned earlier. In orderto investigate the effectiveness of this solution, multipleVLANs were created according to the multicast groups

listed earlier, see table (5).

f*1Uo Z-T_ _ _ _ _ .M .t~_ = %_ _ _ _ _ _

i I I~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

TABLE V. DIVIDING THE NODES INTO DIFFERENT VLANS

VLAN1 SI - Local Controller -AC -AC2-Global Controller-HMI

VLAN2 S2 - Local Controller - AC I - AC2 - Global Controller-HMI

VLAN3 S3-Local Controller - AC1 - AC2 - Global Controller-HMI

VLAN4 SI - Local Controller - AC I - AC2

VLAN5 S2 - Local Controller - ACt - AC2

VLAN6 S3- Local Controller - ACt - AC2

VLAN7 S4 - Local Controller - ACt - AC2-HMI

VLAN8 S5 - Local Controller - ACt - AC2-HMI

VLAN9 S6 - Local Controller - ACt - AC2-HMI

VLAN1O S7 - Local Controller -ACt - AC2-HMI

VLAN1 1 S8 - Local Controller - ACt - AC2-HMI

VLAN12 Local Controller- Global Controller-AC3

The VLANs shown in the table above were submittedto the OPNET environment. The available type ofVLAN inOPNET environment is port based VLAN. The change inlatency is shown in Figure (5) and the traffic map of thenetwork is shown in Figure (6).

Figure 5. Latency Variation in Multicasting/VLAN Case

0(-(

01--

0~~

Figure 6. Traffic Map in Multicasting/VLAN Case

As compared to Figure (4), the traffic on the network isgreatly reduced and the flooding problem was totallyremoved which affects positively on the latency .The othergain beyond using Multicasting / VLAN solution, is theoptimized use of industrial node resources, especially thesensors CPU utilization, which was reduced to (20%).Multicasting frees the nodes from running the unnecessarytasks. At the same time, partitioning the network to multipleVLANs , guides the multicasting traffic to the intendeddestinations only. It is also noted that the FTP responsebecomes (3 Sec. ), which reflects the enhanced abilities ofthe nodes to receive and process more traffic, withoutthreatening their primary tasks.

VII. CONCLUSIONSIn this paper , Industrial Ethernet is studied (using

OPNET simulation package) under different circumstancesof the very hard real time constraints of a substationautomation system. The reference model of the systemassumes the use of the unicast procedures in the generationprocess of the packets. Unicast consumes CPU power andresources of the industrial node. In addition, if the packetproduction rate is high, queuing delay is created at differentnetwork layers of the industrial node. All this adds extradelay to the total value of the latency. Multicastingtechnique is used to decrease the latency by freeing thenodes from running the unnecessary tasks. However,flooding problem could affect on the network delay andhence, latency. Partitioning the network to multiple VLANs, guides the multicasting traffic to the intended destinationsonly and enhance the nodes ability to receive and processmore traffic, without threatening their primary tasks

REFERENCES[1] "Industrial Ethernet: A Control Engineer's Guide",

Cisco Systems, 2001.[2] Potter , " Using Ethernet for Industrial I/0 and Data

Acquisition", National Instruments Corporation 1999.[3] "Understanding and Evaluating Ethernet and TCP/IP

Technologies for Industrial Automation",OPTOCompany, 2001.

[4] R. Daoud ,H. Elsayed ,H. Amer and S. Eid"Performance of Fast and Gigabit Ethernet inNetworked Control Systems," IEEE Proceedings ofInternational MWSCAS, 27-30 December 2003.

[5] Sample Specification: Substation Automation (SA),High-Voltage Transmission Substation, 2003, availableat:www.abb.com

[6] "IEC 61850 - An Overview for Users", available at:www.sisconet.com/downloads,2004.

[7] T. Skeie, S. Johannessen and C. Brunner, "Ethernet inSubstation Automation," IEEE Control Syst. Vol. 22,no. 3, June 2002.

[8] D. Dolezilek , "Case Study Of A Large TransmissionAnd Distribution Substation AutomationProject",Schweitzer Engineering Laboratories,Inc.,Pullman, 1999.

[9] P. Paith ,"transmission Network Protection(Theory andPractice)", MARCEH DEKKER publishing, 1998.

[10] A. Apostolov , " Distributed Intelligence in IntegratedSubstation Protection and Control Systems", ALSTOMCompany, 1999.

[11] "Understanding TCP/IP" , available at:www.techsupportalert.com/pdf/cO4100.pdf,2000.

08

07OX0.6 -

Q 0.5E> 0.4-

x 03-

0.2 -

O1 -

00 50 100 150 200

Simulation time (Sec.)

-Global latency - Local Latenr|

[12]"IP Multicast Technology Overview", Cisco systems,2001.

[13] D. Zaniewski, "Managing Multicast Messages InEthernet/Ip Networks To Deliver Time-Critical Data",Rockwell Automation, Presented at the ODVA CIPNetworks Conference & 10th Annual Meeting,November 16-18, 2004.

[14] A.S. Tanenbaum , "Computer Networks", Prentice-Hall Publishing,2003.

[15]"The Virtual LAN", 3Com Inc, available at:www.3com.com/nsc/200374.html,2002.