IEEE 1588 TIME SYNCHRONISATION AND DATA FLOW ASSESSMENT FOR IEC 61850 BASED POWER TRANSMISSION SUBSTATIONS A thesis submitted to The University of Manchester for the degree of Doctor of Philosophy in the Faculty of Science and Engineering 2020 Mingyu Han School of Engineering Department of Electrical and Electronic Engineering
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
POWER TRANSMISSION SUBSTATIONS
A thesis submitted to The University of Manchester for the degree
of
Doctor of Philosophy
2020
CONTENTS
2
CONTENTS
1.1.1 Standardised substation secondary system based on IEC 61850
................... 18
1.1.2 Obtaining time information for substations
................................................... 21
Research objectives
...............................................................................................
22
Introduction
...........................................................................................................
29
2.2.1 Use of time synchronisation in power systems
.............................................. 30
2.2.2 Timing methods and challenges in delivering accurate time
information ..... 31
Overview of IEEE 1588
........................................................................................
38
2.3.1 Clock types
....................................................................................................
39
2.3.4 Synchronisation principle
..............................................................................
44
2.3.5 PTP profiles
...................................................................................................
49
Summary
...............................................................................................................
61
Introduction
...........................................................................................................
63
Data flow in substation automation networks
....................................................... 69
3.3.1 Main traffic composition and characteristics
................................................. 69
3.3.2 Network architecture
.....................................................................................
72
Traffic management methods
................................................................................
79
Literature survey of substation communication network performance
studies ..... 82
3.7.1 Quantitative assessment of network performance using
simulation tools ..... 83
3.7.2 Evaluation of maximum delay using analytical approaches
.......................... 84
CONTENTS
3
3.7.4 Traffic management strategies to improve communication
performance ..... 87
3.7.5 Issues of message delay / loss and proposed solutions
.................................. 88
3.7.6 Other communication methods for data transfer within the same
substation 90
Summary
...............................................................................................................
91
Testing Philosophy
........................................................................................................
93
Testing methods
..................................................................................................
102
4.3.2 Data latency measurement
...........................................................................
107
4.3.3 Function and configuration of network impairment emulator
..................... 109
4.3.4 Network traffic
injection..............................................................................
111
Summary
.............................................................................................................
114
Assessment of IEEE 1588 for Precision Timing in IEC 61850
Substations 116
Introduction
.........................................................................................................
116
5.2.1 Long-term satellite visibility and synchronisation accuracy
based on different
constellation selections
...............................................................................................
118
5.2.2 Transient characteristics of clocks during loss / restoration
of time reference
123
Use of PTP in various sampled value based substation networks
....................... 127
5.3.1 Accuracy of PTP under various process bus architectures
.......................... 127
5.3.2 Scalability of PTP in a chain connection
..................................................... 130
5.3.3 Synchronisation performance of PTP in complete substation
networks ..... 133
5.3.4 Impact of PTP traffic from multiple time domains
..................................... 138
5.3.5 Slave clock characteristics during Follow_Up message delay
.................... 140
Transient response analysis of PTP synchronisation systems
............................. 142
5.4.1 Effect of BMCA switchover
........................................................................
142
5.4.2 Fault tolerance assessment of highly redundant system
architecture due to
communication link loss
.............................................................................................
145
Protection Applications
..............................................................................................
149
6.2.1 Historical cyber-attack events
......................................................................
150
6.2.2 Typical classification of attacks on IEC 61850 substation
automation systems
151
6.2.3 Threats of time synchronisation attacks on protection and
control applications
153
CONTENTS
4
6.3.1 Theoretical analysis of introduced time errors on PTP
synchronisation
accuracy 154
6.3.3 Attack scenarios and
results.........................................................................
159
Impact of time synchronisation errors on transformer differential
protection .... 165
6.4.1 Mathematical derivation of time errors on transformer
differential protection
scheme 167
6.4.2 Experimental validation using a multi-vendor IEC 61850
testbed .............. 173
Summary
.............................................................................................................
178
180
Challenges in evaluating network latency
...........................................................
184
Experimental evaluation of Ethernet switching behaviours
................................ 186
7.4.1 Laboratory testbed set up
.............................................................................
187
7.4.2 Characteristics of SV and GOOSE residence time in PRP / HSR
enabled
switches 187
7.4.3 Transfer Latency of GOOSE in PRP and HSR process bus networks
with
multi-vendor switches
.................................................................................................
191
7.4.4 Comparison of PTP message handling in one-step and two-step
switches . 194
Visualisation of SV latency in a redundant substation network
.......................... 198
7.5.1 Descriptions of test network
........................................................................
198
7.5.2 Test on seamless transition of SV latency upon communication
link failures
200
Conclusions
.........................................................................................................
209
8.2.1 Time synchronisation related studies
...........................................................
214
8.2.2 Network interactions related studies
............................................................
216
References
....................................................................................................................
217
APPENDIX B Devices Category
............................................................................
235
APPENDIX C Data For Transformer Differential Protection
Operating
Characterises with Time Errors
................................................................................
236
Word count: 57,450
LIST OF FIGURES
LIST OF FIGURES
Figure 1-1: Roadmap of power system protection development from
early 1900s to present
day [5].
..................................................................................................................................
19 Figure 1-2: Structure of IEC 61850 substation secondary systems
[11]. ............................. 21 Figure 2-1: An example of
dedicated timing methods used for synchronising various IEDs
in
a substation automation network [36].
.................................................................................
33 Figure 2-2: An example of networked timing methods (NTP or PTP)
used for synchronising
various IEDs in a substation automation network [36].
....................................................... 34 Figure
2-3: Basic working principle of time synchronisation in a
packet-switched network
[43].
......................................................................................................................................
36 Figure 2-4: Block diagram illustrating how the residence delay
is measured and appended to
the PTP message content in both one-step (a) and two-step (b) TCs
[49], [50]. .................. 40 Figure 2-5: Block diagram
illustrating the architecture of boundary clocks [50].
............... 41 Figure 2-6: State transition of best master
clock algorithm [20]. ......................................... 43
Figure 2-7: Delay request-response mechanism.
..................................................................
46 Figure 2-8: Peer delay request-response
mechanism............................................................
47 Figure 2-9: Designs of test platforms for measuring 1-PPS
offset, (a) Oscilloscope [69], (b)
Measurement server fitted with a PCIe NIC card [20]
......................................................... 51 Figure
2-10: 24 hours 1-PPS pulse offset measurement with various mask
angle settings [20].
..............................................................................................................................................
52 Figure 2-11: Performance of time synchronisation using PTP under
various topologies and
traffic levels [66].
.................................................................................................................
57 Figure 3-1: Interface model representation according to IEC
61850-1:2013 [91]. PROT:
protection, CONTR: control and FCT: function.
.................................................................
65 Figure 3-2: Overall transfer time defined in IEC 61850-5 [12].
........................................... 66 Figure 3-3: GOOSE
transmission characteristic defined in IEC 61850-8-1 [33].
................ 71 Figure 3-4: Ring structure in substation
automation systems [100]. C: control, P: protection.
..............................................................................................................................................
73 Figure 3-5: GOOSE retransmission coexistant with RSTP network
reconfiguration [101]. 73 Figure 3-6: PRP network architecture
[56].
..........................................................................
75 Figure 3-7: PRP Frame compared with normal Ethernet Frame [100].
............................... 75 Figure 3-8: HSR network
architecture with an illustration of duplication discard
mechanism
of multicast messages [56].
..................................................................................................
76 Figure 3-9: HSR frame structure with or without VLAN tagging
[56] ................................ 77 Figure 3-10: Network
interactions at process bus containing SV, GOOSE and PTP traffic
[95].
..............................................................................................................................................
79 Figure 3-11: VLAN frame format compared with standard frame
format [102] ................. 80 Figure 3-12: Arrival and service
curves to determine delay bound (τMAX) and backlog bound
(QMAX) according to Network Calculus theory [137].
.......................................................... 85
Figure 3-13: Testbed arrangement for investigation SV latency in a
PRP process bus network
[144].
....................................................................................................................................
87 Figure 3-14: Concept of SV residence delay measurement [159]
........................................ 90 Figure 4-1: Schematic
representation of the testbed architecture.
........................................ 94 Figure 4-2: Photo of
the test system, (a) time synchronisation rack and monitoring
system, (b)
protection system rack.
.........................................................................................................
96 Figure 4-3: Time servers used in the test systems.
............................................................... 97
Figure 4-4: Slave X and Slave Y used in the test systems.
................................................... 98 Figure 4-5:
Ethernet switches used in the test systems.
........................................................ 98 Figure
4-6: MiCOM relay P643 used for the test systems [175].
......................................... 99
LIST OF FIGURES
6
Figure 4-7: Merging units used in the test systems.
........................................................... 100
Figure 4-8: Traffic generator/ network emulator [178].
..................................................... 100 Figure
4-9: Measurement server [179].
..............................................................................
101 Figure 4-10: Real-time monitoring interface of the measurement
server, showing the 1-PPS
deviation of different clocks compared with the reference signal.
..................................... 101 Figure 4-11:
10/100/1000BASE-T Tap used in the test system [180].
.............................. 101 Figure 4-12: Network Capture
Card with hardware timestamping used in the test system
[181].
............................................................................................................................................
102 Figure 4-13: System set up for validating signal delay caused
by attenuators. ................. 103 Figure 4-14: Signal delay
comparison between normal 1-PPS with 1-PPS after attenuation.
Div: division.
......................................................................................................................
104 Figure 4-15: Test setup for measurement server calibration; (a)
GM A connected to the
reference and input channel 1; (b) GM A connected to the reference,
and GM B connected to
input channels 1 and 2.
.......................................................................................................
105 Figure 4-16: 1-PPS delay variation, (a) between reference and
input channel 1 (b) between
channel 1 and 2.
..................................................................................................................
105 Figure 4-17: Schematic diagram showing the PTP test network
arrangement. .................. 106 Figure 4-18: Test setup for
precisely measuring transfer latency.
...................................... 108 Figure 4-19: Pin
assignment for making the cable to synchronise the Card (only pins
3 and 7
are used for carrying the pulse signals).
.............................................................................
108 Figure 4-20: High precision timestamp recorded by the capture
card. ............................... 108 Figure 4-21: Experimental
Setup for investigating PTP messages impairment on slave
clocks.
............................................................................................................................................
110 Figure 4-22: Configurations of emulator software interface to
achieve message modification.
............................................................................................................................................
110 Figure 4-23: Wireshark network capture showing the Sync message
impairment by inserting
additional values in the correctionField.
............................................................................
111 Figure 4-24: SV format with VLAN settings.
....................................................................
112 Figure 4-25: Captured SV traffic from Wireshark.
............................................................ 112
Figure 4-26: Traffic pattern of 79.8% at 100 Mb/s using a single SV
stream. ................... 113 Figure 4-27: Traffic pattern of
79.5% at 100 Mb/s using multiple SV streams. ................ 113
Figure 4-28: Synthetic SV streams captured by
Wireshark................................................ 114
Figure 5-1 Experimental setup to evaluate the effect of
constellation selection on satellite
visibility and timing accuracy.
............................................................................................
119 Figure 5-2 Long term performance of satellite receivers of
satellite visibility. ................ 120 Figure 5-3 Satellite
trajectory plot on different constellation selections (a) GPS,
(b)
GLONASS, (c) GPS+GLONASS.
.....................................................................................
120 Figure 5-4 3D diagram showing antenna locations with tall
building to the northeast. .... 121 Figure 5-5 Long term
performance of synchronisation accuracy based on different
constellation selections (with reference to GM A)
............................................................. 122
Figure 5-6 1-PPS offset of Grandmaster B and Slave Y (synchronised
to Grandmaster B via
PTP) compared to the reference 1-PPS.
.............................................................................
123 Figure 5-7: Experimental setup for investigating holdover
capability and transient
characteristic of clocks.
......................................................................................................
124 Figure 5-8: Clock holdover capability over a 24 hour monitoring
period with reference to GM
A; (a) all clocks, (b) zoomed view for GM D and Slave Y.
............................................... 125 Figure 5-9:
Resynchronisation of clocks under different time steps with (a)
1µs, (b) 5µs, (c)
10µs.
...................................................................................................................................
126 Figure 5-10: Experimental setup for investigating
synchronisation accuracy under process
bus networks, (a) Star, (b) RSTP, (c) PRP, (d) HSR.
......................................................... 129
Figure 5-11: Sample distribution to compare the accuracy of PTP in
different process bus
architectures.
.......................................................................................................................
130
7
Figure 5-12: System set-up for measuring the inaccuracy of
transparent clocks. .............. 131 Figure 5-13: Pulse
difference distribution for timing accuracy in a Star-connected
network.
............................................................................................................................................
132 Figure 5-14: Experimental setup for investigating
synchronisation accuracy across substation
networks: (a) PRP over HSR, (b) RSTP over HSR and (c) HSR over HSR.
..................... 134 Figure 5-15: Synchronisation performance
of Slave X and Y using highly redundant
topologies............................................................................................................................
136 Figure 5-16: Synchronisation performance of Slaves X and Y in
substation networks in the
presence of GOOSE and SV traffic.
...................................................................................
137 Figure 5-17: Synchronisation performance under multi-domain PTP
traffic ..................... 139 Figure 5-18: Impact of
multi-domain PTP traffic on synchronisation performance (with
reference to GM A).
............................................................................................................
139 Figure 5-19: System arrangement to investigate the impact of
Follow_Up Delay on PTP
synchronisation performance.
.............................................................................................
141 Figure 5-20: Synchronisation performance of (a) Slave X and (b)
Slave Y under Follow_Up
delay impairment (with reference to GM A).
.....................................................................
142 Figure 5-21: Slave clock transient characteristics during BMCA
switchover.................... 144 Figure 5-22: Fault tolerance of
different highly redundant network topologies due to
communication link loss (with reference to GM B).
.......................................................... 146
Figure 6-1: Time synchronisation attacks on a PTP system using E2E
delay measurement
mechanism, (a) Delay attack performed on Sync messages, (b) Packet
modification attack
performed on Sync messages.
.............................................................................................
155 Figure 6-2: Time synchronisation attacks on a PTP system using
P2P delay measurement
mechanism, (a) Delay attack performed on Sync messages, (b) Packet
modification attack
performed on Sync messages.
.............................................................................................
156 Figure 6-3: Experimental setup for performing delay attacks and
packet modification attacks.
............................................................................................................................................
157 Figure 6-4: System setup for excessive traffic inject attack.
.............................................. 158 Figure 6-5:
Testbed setup for PTP spoofing attack.
........................................................... 158
Figure 6-6: PTP delay attack in one-step mode (with reference to GM
A). ....................... 159 Figure 6-7: Delay attack for P2P in
two-step mode (with reference to GM A). ................ 160 Figure
6-8: Packet modification attack on both one-step and two-step (with
reference to
GM
A).................................................................................................................................
161 Figure 6-9: Synchronisation performance of slave clocks under
excessive traffic attack (with
reference to GM A).
............................................................................................................
162 Figure 6-10: PTP spoofing attack with the aid of network
impairment emulator (with
reference to GM A)
.............................................................................................................
163 Figure 6-11: Schematic diagram comparing conventional and IEC
61850 based transformer
differential protection scheme.
...........................................................................................
166 Figure 6-12: (a) Typical Bias Characteristic for transformer
differential protection, (b) circuit
diagram to illustrate the measured current. 87T: ANSI code for
transformer differential
protection
............................................................................................................................
168 Figure 6-13: Theoretical analysis of differential protection
under external faults with time
synchronisation errors.
........................................................................................................
170 Figure 6-14: Theoretical analysis of the actual restraint curve
with synchronisation errors.
............................................................................................................................................
173 Figure 6-15: Experimental setup for transformer differential
protection scheme that uses
sampled value process bus.
.................................................................................................
174 Figure 6-16: Working principle of the Emulator to control the
time offset. ...................... 175 Figure 6-17: Performance of
differential protection under external faults with added time
errors, (a) full range, (b) zoomed view.
..............................................................................
176
LIST OF FIGURES
8
Figure 6-18: Effect of time error on the operating characteristics
compared with the restraint
curve.
..................................................................................................................................
178 Figure 7-1: Transfer latency and Network latency requirement.
........................................ 182 Figure 7-2: Overview
of composition of transfer latency.
................................................. 184 Figure 7-3:
Challenges in evaluating network latency due to increasing network
sizes, various
ports speeds, introductions of various redundancy protocols and
complex traffic interactions.
............................................................................................................................................
186 Figure 7-4: Test bed set up for measuring the network latency.
........................................ 187 Figure 7-5:
Illustration of test scenarios for investigating switching
behaviours. .............. 188 Figure 7-6: illustration of
relationship between packet size and combined delay, (a) Switch
H,
(b) Switch R.
.......................................................................................................................
190 Figure 7-7: Analysing average network latency for GOOSE
messages (409 bytes) under PRP
or HSR architectures. The type of port used for connecting the
switches is shown in the
brackets.
..............................................................................................................................
192 Figure 7-8: Data flow path and latency measurement of SV
message stream in HSR-HSR
topology under various communication loss conditions.
................................................... 200 Figure 7-9:
Website based configuration interface of Redboxes
........................................ 201 Figure 7-10:
Purpose-made serial cable for setting up data connection between
switches and
workstation.
........................................................................................................................
202 Figure 7-11: Command Line Interface console window for switch
configuration via serial
connection.
..........................................................................................................................
202 Figure 7-12: Preliminary test results comparing different
methods that simulate
communication link failures.
..............................................................................................
203 Figure 7-13: Comparison of network traffic between CLI console
and website based interface.
............................................................................................................................................
204 Figure 7-14: Visualisation of real-time SV data latency
measurement in HSR-HSR network
during various communication link losses and traffic conditions.
..................................... 205 Figure 8-1: Test setup by
Endace to examine the PTP synchronisation accuracy in the
WAN,
considering asymmetrical link delays, heavy loading conditions and
contention (collision)
[222].
..................................................................................................................................
214
LIST OF TABLES
Table 2-1: Time synchronisation classes specified in IEC 61850-5
[12]. ............................ 32 Table 3-1: Transfer time
classes and corresponding requirement according to IEC
61850-5
[12].
......................................................................................................................................
67 Table 3-2: Examples of recovery time requirement
[12]...................................................... 69 Table
3-3: Summary of main process bus traffic and transmission time
requirement, modified
based on the published data in [96] and IEC 61850-5 and IEC
61850-90-4 [12], [97]. ....... 72 Table 4-1: Voltage level after
attenuation. The colour coding means the voltage level of
attenuated 1-PPS signal is acceptable.
................................................................................
104 Table 4-2: Statistics of measurement error of 1-PPS measurement
server......................... 105 Table 4-3: Statistical of
measures of error between two identical Sync messages .............
109 Table 5-1 24-hour synchronisation accuracy of GM B under
different constellations (with
reference to GM A)
.............................................................................................................
122 Table 5-2 Summary of synchronisation accuracy in process bus
networks. ...................... 130 Table 5-3 Summary of
synchronisation accuracy in a Star-connected network.
............... 133 Table 5-4 Summary of traffic type and bandwidth
............................................................. 135
Table 5-5 Summary of synchronisation accuracy in a highly redundant
substation
architectures.
.......................................................................................................................
137 Table 7-1: Summary of residence time of for SV and GOOSE in
different test scenarios. 189 Table 7-2: Summary of the combined
delay in different test scenarios. ............................ 189
Table 7-3: Relationship between residence time and packet size for
both switches and
estimate residence time for 409 bytes GOOSE.
.................................................................
193 Table 7-4: Comparison between test and estimated residence time
for 409 byte GOOSE
across PRP and HSR process bus networks.
......................................................................
194 Table 7-5: Summary of residence time of Sync messages in
one-step TC (Switch R) under
various traffic conditions.
...................................................................................................
196 Table 7-6: Summary of residence time of Follow_Up messages in
one-step TC (Switch R)
under various traffic conditions.
.........................................................................................
196 Table 7-7: Summary of residence time of Sync messages in
two-step TC (Switch M) under
various traffic conditions.
...................................................................................................
197 Table 7-8: Summary of residence time of Follow_Up messages in
two-step TC (Switch M)
under various traffic conditions.
.........................................................................................
197 Table 7-9: Summary of SV data latency measurement in HSR-HSR
network during various
communication link losses and traffic conditions.
............................................................. 206
Table A-1: Comparison Summary of main differences between PTP
profiles that are used for
substation automation systems
...........................................................................................
233 Table B-1: List of equipment in the laboratory testbed
...................................................... 235 Table
C-1: Mathematical calculation of transformer differential
protection operating
characteristics under 250 µs time synchronisation error.
................................................... 236 Table C-2:
Mathematical calculation of transformer differential protection
operating
characteristics under 500 µs time synchronisation error.
................................................... 238 Table C-3:
Mathematical calculation of transformer differential protection
operating
characteristics under 750 µs time synchronisation error.
................................................... 240 Table C-4:
Mathematical calculation of transformer differential protection
operating
characteristics under 1000 µs time synchronisation error.
................................................. 242
LIST OF ABBREVIATIONS
9-2 LE IEC 61850-9-2 Light Edition
A/D Analogue-to-Digital
b/s Bit Per Second
Base Profile IEC/IEEE 61850-9-3 Precision Time Protocol Profile For
Power
Utility Automation
CLI Command Line Interface
DFR Digital Fault Recorders
DoS Denial of Service
GM Grandmaster
GPS Global Positioning System
HMI Human Machine Interface
HSR High-Availability Seamless Redundancy
IEC International Electrotechnical Commission
IED Intelligent Electronic Device
IFG Inter-Frame Gap
IP Internet Protocol
ISG Inter-Stream Gap
IT Information Technology
NIC Network Interface Controller
NIST National Institute of Standards And Technology
NTP Network Time Protocol
OMNeT++ Objective Modular Network Testbed In C++
OPNET Optimized Network Engineering Tool
OSI Open System Interconnection
OTN Optical Transport Network
P2P Peer-to-Peer
PDC Phasor Data Concentrators
PDV Packet Delay Variation
PLC Programmable Logic Controllers
PMU Phasor Measurement Unit
Power Profile v1 IEEE C37.238-2011 Standard Profile For Use Of IEEE
1588
Precision Time Protocol In Power System Applications
Power Profile v2 IEEE C37.238-2017 Standard Profile For Use Of IEEE
1588
Precision Time Protocol In Power System Applications
PRP Parallel Redundancy Protocol
PTP Precise Time Protocol
Synchronization Protocol For Networked Measurement And
Control Systems
PTP v2
Synchronization Protocol For Networked Measurement And
Control Systems
PTP v2.1
Synchronization Protocol For Networked Measurement And
Control Systems
RX Receive
SDH Synchronous Digital Hierarchy
SDN Software Defined Networking
SFD Start Frame Delimiter
SONET Synchronous Optical Networking
TDoS Telephony Denial-of-Service
UDP User Datagram Protocol
UPS Uninterruptible Power Supply
UTC Coordinated Universal Time
VT Voltage Transformer
WAN Wide Area Network
ABSTRACT
13
ABSTRACT
IEEE 1588 Time Synchronisation and Data Flow Assessment for IEC
61850 based Power
Transmission Substations
Degree: Doctor of Philosophy Date: Dec 2020
Substation automation systems (SAS) in accordance with IEC 61850
are gaining popularity
in the power industry and is expected to dominate the design of
substations worldwide in the
near future. The standard offers a ‘future-proof’ solution for the
protection, automation and
control (PAC) systems with cost reduction in design, operation and
maintenance. IEEE 1588
Precision Time Protocol (PTP) is widely considered for use in IEC
61850 based substations
due to its achievable sub-microsecond accuracy over Ethernet, which
satisfies the accuracy
required by sampled value (SV) process bus and other advanced PAC
applications. However,
PTP is still not considered by many utilities to be a fully-mature
technology for reliable time
distribution in substations. The propagation delay variations of
PTP messages caused by
network traffic are sometimes suspected of resulting in
unacceptable timing errors. Substation
engineers also require sufficient understanding of the PTP timing
principle and the
characteristics of network devices to operate and maintain the
system. Additional
investigations are needed urgently to study the characteristics of
communication network
behaviours in the presence of all principle IEC 61850 traffic.
Confidence must be given to
the system operators that the intrinsic data flow interactions in
the well-designed substation
automation system does not degrade the performance of either time
synchronisation or other
PAC applications.
This research focuses on the use of PTP for precision timing, and
interactions between multi-
protocol substation traffic. A hardware test platform consisting of
commercially-available
products (e.g. clocks, Ethernet switches, merging units and
protection relays) was designed
and built to explore the performance both at the individual
component level and at the
complete substation automation system level. Testing tools
including a traffic generator /
network impairment emulator, a time synchronisation measurement
server, an OMICRON
test set, a network capture card and an Ethernet tap were
integrated into the test platform to
simulate several network conditions in both electrical and data
communication systems.
These network conditions were critical to fully understand the
network characteristics and
identify performance boundaries.
The results systematically demonstrate that it is feasible to
implement PTP as an alternative
for conventional dedicated timing systems. PTP can be used in
conjunction with seamless
redundancy protocols as defined in IEC 62439-3 to enhance the
system reliability while still
maintaining a ±1 µs accuracy requirement. Meanwhile, care must be
taken with system
vulnerabilities discovered under time synchronisation attacks.
Delay, modification, denial of
service and systematic spoofing attacks can result in deterioration
of synchronisation
accuracy. A transformer differential protection scheme was selected
as an example to
evaluate, theoretically and experimentally, the effect of
inadequate time synchronisation on
operating characteristics. The transfer latency of various process
bus data including PTP, SV
and Generic Object Oriented Substation Event (GOOSE) is studied
using in-service Ethernet
switches supporting seamless redundancy protocols. The obtained
data transfer
characteristics can be used to enhance the accuracy of latency
estimation.
Using the hardware testbed with “live” substation equipment
discovers factors that often
struggle to be found in simulation and analytical studies. The
findings of this research can be
used as a performance reference which is valuable for various
stakeholders including the
standard board, power utilities, manufacturers and other academic
researchers.
DECLARATION
14
DECLARATION
No portion of the work referred to in the thesis has been submitted
in support of an application
for another degree or qualification of this or any other university
or other institute of learning.
COPYRIGHT STATEMENT
COPYRIGHT STATEMENT
i. The author of this thesis (including any appendices and/or
schedules to this thesis) owns
certain copyright or related rights in it (the “Copyright”) and
s/he has given The University
of Manchester certain rights to use such Copyright, including for
administrative purposes.
ii. Copies of this thesis, either in full or in extracts and
whether in hard or electronic copy,
may be made only in accordance with the Copyright, Designs and
Patents Act 1988 (as
amended) and regulations issued under it or, where appropriate, in
accordance Presentation
of Theses Policy You are required to submit your thesis
electronically Page 11 of 25 with
licensing agreements which the University has from time to time.
This page must form part
of any such copies made.
iii. The ownership of certain Copyright, patents, designs,
trademarks and other intellectual
property (the “Intellectual Property”) and any reproductions of
copyright works in the thesis,
for example graphs and tables (“Reproductions”), which may be
described in this thesis, may
not be owned by the author and may be owned by third parties. Such
Intellectual Property
and Reproductions cannot and must not be made available for use
without the prior written
permission of the owner(s) of the relevant Intellectual Property
and/or Reproductions.
iv. Further information on the conditions under which disclosure,
publication and
commercialisation of this thesis, the Copyright and any
Intellectual Property and/or
Reproductions described in it may take place is available in the
University IP Policy (see
http://documents.manchester.ac.uk/DocuInfo.aspx?DocID=2442 0), in
any relevant Thesis
restriction declarations deposited in the University Library, The
University Library’s
regulations (see http://www.library.manchester.ac.uk/) and in The
University’s policy on
Presentation of Theses.
ACKNOWLEDGEMENT
16
ACKNOWLEDGEMENT
First and foremost, I am immensely grateful to my supervisors,
Prof. Peter Crossley and Prof.
Zhongdong Wang, and advisor, Dr. Qiang Liu, for their technical
guidance, invaluable advice,
insightful comments, and continuous support during my PhD study. I
would not be able to
come this far without their encouragement and inspiration.
I would like to acknowledge EPSRC Centre for Doctoral Training in
Power Networks and
The University of Manchester for sponsoring my PhD study. I would
like to thank CIGRE
UK B5 committee for accepting me as a UK technical member to
contribute to CIGRE
International Working Group B5/D2.67.
I would like to express my special thanks to Dr. Hao Guo, Dr.
Yucong Zhao, Dr. Wentao Zhu,
Dr. James Hill in Ferranti Building and Mr. Sajjid Salyani from
Siemens plc for co-authoring
the papers. My sincere gratitude also goes to Mr. Stephen Potts,
Dr. Nan Liu, Prof. Xiaolong
Chen, Dr. Hasan Uppal and many other colleagues and friends in
Ferranti for their advice,
encouragement and support to help me ride through many stressful
moments.
I would like to thank Siemens plc, UK for kindly providing Ethernet
switches for this project,
and Hirschmann, Tekron, Oregano Systems, and Meinberg for their
prompt and valuable
technical discussions.
As one of my proudest achievements alongside my PhD study, I am
very grateful to former
Chairmen Dr. James Hill and Mr. Trevor David for their leadership
and foundation in the
IEEE PES Student Branch Chapter, and to Dr. Angeliki Loukatou and
Ms. Siwei Liu for their
immense contribution during my chairmanship of the committee. Those
prestigious awards
we have won will never be forgotten.
Last but not least, I would like to express my deepest gratitude to
my family, especially my
parents and my wife, Mrs. Tong Yu. Their understanding, patience
and unconditional love
are my greatest driving force to the perfection. The sacrifices
Tong made for travelling
thousands of miles to Manchester after six years of long distance
relationship to accompany
me to complete my PhD study will always be appreciated from the
bottom of my heart. I hope
my accomplishment will make my family proud.
LIST OF PUBLICATIONS
LIST OF PUBLICATIONS
[1] M. Han, H. Guo, and P. Crossley. "IEEE 1588 time
synchronisation performance for IEC
61850 transmission substations." International Journal of
Electrical Power & Energy
Systems 107 (2019): 264-272.
[2] W. Zhu, M. Han, J. V. Milanovic, and P. Crossley. "Methodology
for Reliability
Assessment of Smart Grid Considering Risk of Failure of
Communication Architecture."
IEEE Transactions on Smart Grids. vol. 11, no. 5, pp. 4358-4365,
Sept. 2020, doi:
10.1109/TSG.2020.2982176.
[3] M. Han, P. Crossley, S. Salyani, “Evaluation of Process Bus
Data Latency with
PRP/HSR Enabled Ethernet Switches,” in The 15th International
Conference on
Developments in Power System Protection (DPSP) 2020.
[4] M. Han and P. Crossley, "Vulnerability of IEEE 1588 under Time
Synchronization
Attacks," 2019 IEEE Power & Energy Society General Meeting
(PESGM), Atlanta, GA,
USA, 2019, pp. 1-5, doi: 10.1109/PESGM40551.2019.8973494.
[5] M. Han, Y. Zhao, and P. Crossley, “Impact of Time
Synchronisation Errors on
Transformer Differential Protection that uses the IEC 61850 Process
Bus,” in CIGRE
Chengdu Symposium, 2019.
[6] M. Han, and P. Crossley, “Performance Evaluation of IEEE 1588
for Precision Timing
in IEC 61850 Substations,” in CIGRE Colloquium 2019, Tromso
Norway.
[7] M. Han, J. Hill, Z. Wang, and P. Crossley, “Thermal Evaluation
of Railway Transformer
Used in Autotransformer Feeding Systems,” in IEEE International
Energy Conference
(ENERGYCON), 2018.
1.1.1 Standardised substation secondary system based on IEC
61850
Substations play a vital role in the electrical power system as
they not only convert voltage
levels but also provide switching and protection functions. Inside
conventional substations,
protection and control devices are connected point-to-point using
hardwired systems, and
signals such as measurements, signals and commands traverse over
copper wires to the
control rooms [1]. With growing electricity demand and increasing
voltage levels for
transmission, the size of substations increases to accommodate
bulkier primary equipment
and the associated secondary system devices [2]. This implies
substations which opt for
conventional schemes will require larger cabling systems, raising
costs and creating
difficulties in installation and maintenance.
Meanwhile, communication used for substation protection, automation
and control (PAC)
system has seen technical advances over the past century; and the
world has seen a dramatic
evolution in how the electricity grid is operated [3]. At the
beginning of 1900s, protection
was realised using electro-mechanical based relays. During this
time, information such as line
loading and other operational commands can only be communicated via
telephone [4]. Later
in 1930s, some remote control devices that could communicate via
telephone-switching
network became available to read grid status and realise control
operations [4]. With the
application of electronic and computer technologies after 1960s,
processing capability and
communication bandwidth is no longer a limiting factor for PAC
systems. Hence, protection
CHAPTER 1: INTRODUCTION
relays have enjoyed a quick development, from electro-mechanical to
integrated circuit and
micro-computer based technologies [5].
This blooming of communication capability was accompanied by
various theoretical
breakthroughs in protection and control algorithms. A roadmap of
the development of
protection systems is illustrated in Figure 1-1. With the
integration of communication services,
modern protection schemes are not restrained to overcurrent,
distance or differential
protection. Centralised, adaptive, or even artificial-intelligent
based protection algorithms
have been introduced since the 1970s. Furthermore, conveying
information beyond the
substation domain using wired or wireless technologies unlocks more
opportunities for
advanced wide-area PAC in future power systems [6].
Power system operation is in constant evolution, which creates new
challenges for PAC
systems. Countries, such as the UK, are committed to reduce carbon
emissions, and the UK
National Grid aims to achieve a net-zero emission electricity
system by 2025 [7]. The
increased penetration of renewable energy resources reduces system
fault levels and poses
challenges to conventional protection relays [8]. One of the key
solutions is to have a better
system status awareness with faster fault identification and
clearance, which cannot be
achieved without fast, secure and reliable communication.
Figure 1-1: Roadmap of power system protection development from
early 1900s to present
day [5].
Traditionally, many communication infrastructures in substations
remained proprietary.
However, the life-time of secondary systems is significantly
shorter than their associated
primary plants. For example, micro-processor based numerical relays
are expected to have
in-service lifetime of around 15 years, compared to 40-70 years for
the primary plant.
Moreover, due to the rapid development of information and
communication technology (ICT),
future renewal of substation secondary infrastructure becomes a
huge issue as the newly
added devices may not be compatible with existing ones [9].
Consequently, the power
industry required a standardised infrastructure that could provide
cheap, reliable,
interoperable and sustainable communication for delivering and
managing information [10].
IEC 61850 was introduced by International Electrotechnical
Commission (IEC) Technical
Committee 57 as a guideline to standardise the engineering process
in order to enable
intelligent electronic devices (IEDs) from various vendors to
communicate with each other.
The standard proposes a local area network (LAN) using Ethernet to
replace the hardwired
cabling system. IEC 61850 defines three levels in the substation
automation system (SAS)
architecture, namely process level, bay level, and station level as
shown in Figure 1-2. The
data exchange between different levels is realised using so called
station bus and process bus.
In an IEC 61850 based substation that uses the process bus, the
analogue voltage and current
measurement signals are digitised as sampled values (SVs). These
digital measurement values
are transmitted via optic fibres in the substation secondary
Ethernet communication networks,
which can be subscribed to by the IEDs for various protection and
control applications.
Process buses increase the flexibility and functionality of the
substation secondary systems
and make future expansion and retrofitting easier to implement [9].
Ethernet communication
offers immediate reduction of cabling cost, and removes the safety
hazards associated with
having open circuits connected to instrument transformers in the
relay control room [1]. In
addition, the standardised system allows utilities to buy products
from different vendors with
“plug and play” features. Furthermore, availability of voltage /
current measurements is
greatly enhanced by multicasting over Ethernet compared to
point-to-point connection in the
conventional substations. This is more suitable for implementing
advanced PAC functions.
CHAPTER 1: INTRODUCTION
Figure 1-2: Structure of IEC 61850 substation secondary systems
[11].
1.1.2 Obtaining time information for substations
Although IEC 61850 provides standardised information exchange
between PAC devices,
without having a common notion of time, a subscribing device cannot
correctly coordinate
the sequence of events to reproduce the actual power system status.
Many applications require
time synchronisation in order to achieve the desired performance,
e.g. data acquisition units
such as merging units (MUs), transmission of Generic Object
Oriented Substation Events
(GOOSE) messages, protection relays, and fault recorders. Losing
time synchronisation may
affect the accuracy of fault diagnosis, decision making, and
automatic PAC devices
operations. The level of required accuracy depends on applications
which are defined
comprehensively in IEC 61850-5 [12].
The methods of obtaining time information by substations have been
adapted to the
development of the available technology. As digital technology
progressed, it was recognised
that information recorded at different sites would be more valuable
if they were synchronised
to a common time [13]. Historically, many countries used
territorial wireless radio
transmitters to synchronise multiple substations with broadcasting
time codes. For example,
CHAPTER 1: INTRODUCTION
22
Rugby-MSF, maintained by the National Physical Laboratory, was used
to cover the entire
UK with achievable accuracy of around ±1 ms [14]. The signal is
transmitted continuously at
60 kHz. However, such low frequency signals suffers from
interference, particularly close to
high voltage plant [13].
Since the early 1990s, distributed Global Positioning System (GPS)
receivers have been
deployed in the power industry for more accurate time
synchronisation [15]. Clocks with
built-in GPS receiver chips can obtain timing accuracy better than
± 100 ns [16]. There are
also other types of Global Navigation Satellite Systems (GNSS)
available such as GLONASS,
Galileo and BeiDou. It is possible to have a signal receiver that
is synchronised to multiple
GNSS signals to enhance satellite visibility. More recently, the
Optical Transport Network
(OTN) is considered to transmit highly accurate time information
across a large geographic
area to provide additional time reference sources at substations
[17]. Integrating such wide-
area time references helps detect intentional or unintentional GNSS
signal interference and
improves overall reliability [18].
High accuracy time distribution inside substations is traditionally
achieved via dedicated
cabling systems carrying one-pulse-per-second (1-PPS) or
inter-range instrumentation -
Group B (IRIG-B) time code. Only very few devices equipped with
GNSS receivers may be
able to get directly time infeed from roof-mounted antennae.
However, the recently
introduced IEEE 1588 Precision Time Protocol (PTP) offers a much
simpler solution by
transmitting time over Ethernet with an accuracy better than ± 1 µs
[19]. Benefits of PTP can
be found in both technical and economic aspects and these will be
extensively reviewed in
Chapter 2. Using PTP as the primary time synchronisation method in
conjunction with a small
number of 1-PPS / IRIG-B systems for legacy devices has become a
trend for future
implementation of IEC 61850 based substations [20].
Research objectives
The power system of today is becoming more complex and dynamic,
requiring more
advanced PAC applications to ensure reliable operation of the power
grid. This requirement,
together with computer and communication technology development,
foster the birth of
IEC 61850 [21]. Standardised PAC systems based on IEC 61850 help to
maintain flexibility
and functionality in the long run [9]. Transmitting time
information across the substation
using dedicated timing systems based on legacy 1-PPS / IRIG-B is no
longer an economical
and future-proof solution. PTP entered the time synchronisation
arena in 2002 and the power
CHAPTER 1: INTRODUCTION
23
industry has shown interest in adopting these technologies to take
the advantage of Ethernet
communication systems while achieving sub-microsecond
accuracy.
However, the deployment of PTP remains slow. Currently in the UK,
PTP is only being
applied at the Wishaw 275 kV substation in Scotland [22].
Transmitting time information
over Ethernet is still not considered to be fully reliable and
secure by some utilities; the fear
of not having tangible point-to-point connections as with the
conventional dedicated timing
methods makes the power industry hesitant in moving to full
Ethernet communication. More
importantly, the in-service performance of PTP in substation
Ethernet systems is still largely
unknown. The achievable accuracy with various substation
communication system structures
and traffic conditions remains to be answered. In particular,
seamless redundancy protocols
based on IEC 62439-3, namely the Parallel Redundancy Protocol (PRP)
and the High-
availability Seamless Redundancy (HSR), are widely considered in
substation secondary
system design as they offer zero-failover time. The performance
assessment of PTP in
complex substation network architectures formed from PRP and HSR
are rarely seen in
literature.
Additionally, interruption of satellite signals or PTP message
transmission is likely to cause
degradation of synchronisation accuracy. More investigations need
to be carried out under
network contingency situations to reveal the transient performance
of commercially-available
PTP products. A comprehensive understanding of the system dynamic
behaviours is essential
to better design the substation automation system architecture and
select the suitable products
to optimise the timing performance.
Furthermore, cyber threats to the PTP system are emerging as the
interconnected Ethernet
system inevitably provides more access points to malicious
attackers. This results in security
concerns over time synchronisation systems based on PTP. To
identify the vulnerability, the
real-time response of individual devices and the entire system
during various forms of attacks
must be assessed in detail. Further again, recognition of the
impact of resultant time errors on
PAC applications will be crucial for the IEC 61850 substations to
defend against potential
time synchronisation attacks.
On the other hand, the packet transmission latency over Ethernet is
non-deterministic, and is
mainly governed by several factors such as communication bandwidth,
network architecture
and queuing delays induced by background traffic. Some
time-critical messages, e.g. SV and
GOOSE, require fast transmission time to meet the performance
requirement specified in IEC
61850-5. However, the latency requirement might not be achieved in
a large substation
CHAPTER 1: INTRODUCTION
congested by multi-protocol messages. Delays or even failures in
delivering time-critical
messages may lead to degradation of PAC performance. Integration of
PTP into the
communication systems brings additional network traffic, making the
traffic interactions
much more complex. End-to-End (E2E) transfer latency is not a new
research topic but most
previous studies were performed using simulations and analytical
models. The results are
normally as good as the parameters and network conditions that was
assumed. The lack of
realistic data often leads to large errors between simulation and
testing. A real-time
assessment of transfer latency of various substation traffic is
urgently required to fully explore
the traffic interactions in IEC 61850 substations.
The ultimate goal of this project is to help the power industry
gain insights into PTP time
synchronisation and data flow characteristics in multi-function
substation communication
networks. Results presented in this thesis can be used to provide
technical recommendations
for designing a more reliable and secure timing architecture for
IEC 61850 based substations.
Specific objectives are formed as follows:
To design and build a multi-vendor hardware testbed for
investigating the PTP
synchronisation accuracy in IEC 61850 substations with various
loading conditions
caused by background traffic.
To evaluate multiple time synchronisation architectures for a
reliable time
distribution across process bus and complete substation
networks.
To understand the PTP time synchronisation performance under
steady-state
conditions and investigate dynamic behaviours of the timing system
under
contingency situations.
To identify the vulnerability and dynamic responses of PTP system
under malicious
time synchronisation attacks.
To analyse the dependency of time synchronisation on PAC
applications, e.g.
transformer differential protection.
To characterise traffic interactions between PTP and other message
protocols in the
same network, and analyse the transfer latency of various process
bus traffic (e.g. SV,
GOOSE and PTP).
CHAPTER 1: INTRODUCTION
25
Contributions
The main contributions of this research are summarised in the
following bullet points. The
research outputs have also been published in various journals and
international conferences.
Design and build a multi-vendor hardware testbed based on IEC 61850
and PTP
with various testing methods
The innovative testbed described in Chapter 4 simulates a small
section of an IEC
61850 transmission substation with PTP being the primary time
synchronisation
method. This testbed exceeds the capability of most conventional
laboratory facilities
by combining PTP time synchronisation with IEC 61850 process bus
functions in a
signal platform. Integrating multi-vendor devices (e.g. clocks,
Ethernet switches,
protection IEDs and MUs) into the testbed also provide a rare
opportunity to evaluate
the system interoperability and cross-compare the performance of
individual devices.
In addition, various testing methods, e.g. 1-PPS offset comparison,
network traffic
injection, network impairment emulation, and precise transfer
latency measurement
and secondary voltage / current injection from Omicron test set,
have been designed
and utilised for the research. These methods allow the experiment
to be conducted in
a controlled and repeatable way so that the in-service behaviours
of individual
component or the entire systems can be studied thoroughly.
Comprehensive real-time performance evaluation of PTP time
synchronisation
Substantial experiment has been carried out to evaluate the
real-time PTP time
synchronisation in the substation Ethernet networks in Chapter 5.
The uniqueness of
this contribution compared to the existing literature is the use of
GNSS in conjunction
with IEC 62439-3 seamless redundancy protocols to enhance the
overall PTP system
reliability. Meanwhile, the achievable synchronisation accuracy of
PTP in digital
substations from station level (time receivers) to process level
(slave clocks) is
investigated in detail, covering both steady-state and transient
scenarios. The level of
comprehensiveness and technical depth are very rare to be found in
the published
literature.
Identification of vulnerability of PTP system under time
synchronisation attacks
The vulnerability analysis described in Chapter 6 is one of the
pioneering studies of
cyber-security threats existed in PTP systems under time
synchronisation attacks. The
impact of attacks is firstly theoretically analysed with
mathematical derivations. After
that, several attack scenarios have been performed on commercial
PTP devices from
different manufacturers. Therefore, the real-time response of PTP
to the time
CHAPTER 1: INTRODUCTION
26
synchronisation attacks can be revealed. This study also takes into
account the PTP
system configurations, which are applicable to the digital
substations so that the
results can be directly beneficial to the power industry and fill
the research gaps.
Some dynamic characteristics of PTP system during real-time attacks
are valuable to
many stakeholders (utilities, manufacturers, standard boards),
which cannot be easily
foreseen using simulation or mathematical analysis.
Investigations of the impact of inadequate time synchronisation on
transformer
differential protection
The dependency of transformer differential protection on
synchronisation accuracy is
investigated both theoretically and experimentally in the second
half of Chapter 6. In
actual substations, inadequate time synchronisation may result from
communication
network contingencies or even malicious cyber-attacks. The
theoretical analysis
presented in this research reveals how the actual bias operating
characteristics will
deviate from the relay setting due to introduced phase errors.
Moreover, a novel
method is proposed to precisely control the time deviation between
MUs in the
testbed. Hence, the impact of time errors on the protection
operation can be quantified.
This method is also applicable to other studies that require the
creation of time
synchronisation errors on PAC devices.
Assessment of transfer latency of process bus traffic in
multi-protocol substation
automation networks
This research carries out a unique analysis of the characteristics
of PRP/HSR enabled,
PTP-aware switches handling SV, GOOSE and PTP data latency. Almost
all existing
simulation and analytical methods for latency estimation lack the
representation of
actual devices, making them less accurate to be used in the real
substation studies.
The experimental results (presented in Chapter 7) obtained based on
the
commercially-available switches from different manufacturers
greatly enhance the
accuracy of latency estimation. In addition, the real-time
measurement of SV latency
in a state-in-the-art highly redundant substation network provides
insights of data
interactions in a multi-functional substation communication
network, which has not
been seen previously in other literature.
CHAPTER 1: INTRODUCTION
Thesis outline
This thesis contains eight chapters and the linkage is showed as
follows.
Chapter 1 introduces the background, discusses the objectives and
summarises the main
contributions of this research.
Chapter 2 provides an overview of high accuracy time
synchronisation in substation
automation systems. Time distribution methods including dedicated
timing (1-PPS and IRIG-
B time code) and networked timing (Network Timing Protocol and PTP)
are reviewed and
compared. A comprehensive review of PTP is also presented covering
clock types,
grandmaster clock election, delay measurement mechanism and various
PTP profiles. A
literature survey is produced on PTP performance evaluations to
help identify the current
research gaps.
Chapter 3 reviews substation communication networks based on IEC
61850. The chapter
first discusses the functions, message types, and communication
requirements of IEC 61850.
This is followed by an introduction of traffic composition and data
flow characteristics in the
substation communication networks. This chapter also addresses the
increasing demand of
system redundancy by going through some commonly used redundant
structures applicable
to IEC 61850 substations. In addition, the challenges of analysing
sophisticated traffic
interactions are discussed, highlighting the importance of having a
well-designed network
with appropriate traffic management methods. A literature survey is
provided on the previous
performance studies of substation communication systems to conclude
this chapter.
Chapter 4 focuses on the design and implementation of the
laboratory test systems with
various testing philosophies. The architecture of the testbed is
presented and individual
devices are described in detail. The experimental methods applied
in this research, including
timing accuracy measurement, network impairment and traffic
injection, are also discussed.
Chapter 5 presents the design and performance evaluation for using
GNSS and PTP in
transmission substations. The content of this chapter forms two
published items, [1] and [6]
in the List of Publications. The results presented in this chapter
provides a comprehensive
overview of the synchronisation accuracy achievable with
commercially available devices in
various network topologies and network conditions. This chapter
also examines the transient
behaviours of grandmaster clocks, and investigates the feasibility
of using multi-GNSS
constellation grandmaster clocks to deliver a more reliable time
reference.
CHAPTER 1: INTRODUCTION
28
Chapter 6 considers the vulnerability of PTP networks and the
impact of inadequate time
synchronisation accuracy on transformer differential protection.
The content of this chapter
forms two published items, [4] and [5] in the List of Publications.
This chapter first illustrates
the potential threats that exist in PTP timing systems using
mathematical analysis; the impact
of delay attacks and modification attacks are quantified. A test
system is designed and
deployed to perform various “mimic” time synchronisation attacks,
which identifies the
characteristics of PTP devices under attack. After this, a
theoretical analysis and experimental
validation of the impact of time synchronisation deviations on
transformer differential
protection are discussed. The experiment is performed using a
hardware testbed which
integrates PTP timing functions into the testing of transformer
differential protection.
Chapter 7 addresses the process bus data latency in a
multi-protocol traffic environment. The
content of this chapter forms two published items, [2] and [3] in
the List of Publications. The
chapter starts with an overview of the composition of transfer
latency and current challenges
in the latency evaluation studies. An assessment of the network
latency of SV, GOOSE and
PTP is carried out using in-service Ethernet switches from
different manufacturers. A real-
time latency measurement of SV messages across a HSR-HSR network is
presented to
visualise the seamless transitions of SV transfer during network
contingencies.
Chapter 8 concludes the findings of this research and provides an
outlook of future research
directions on both time synchronisation and network related
topics.
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
29
Introduction
It is a growing trend to roll out IEC 61850 and PTP in future
substation communication
networks. Time synchronisation, data transmission, protection and
control can all be achieved
in the same Ethernet network. The IEC 61850 family standardises the
engineering process
such that devices in substations can communicate with each other,
regardless of their make,
model or manufacturer. IEEE 1588 has become part of IEC 61850
standard series as
IEC/IEEE 61850-9-3 for power utility automation applications. The
“plug and play” features
that come with IEC 61850 standard series also provide confidence to
the power industry that
the automation systems of today can cope with changing requirements
and technology
evolutions of the future.
This chapter provides a comprehensive overview of highly accurate
time-synchronised
substation networks based on PTP. Use of timing synchronisation
methods with a particular
emphasis of implementation challenges are given in Section 2.2. An
overview of PTP is
presented in Section 2.3, including clock types, one- or two-step
operation modes, delay
measurement mechanisms and the configuration / performance guidance
based on various
PTP profiles. After that, Section 2.3.6 provides a review of the
literature on experimental
evaluation on PTP performance conducted by various researchers.
Factors that can contribute
to synchronisation inaccuracies are categorised into both steady
and transient state and are
discussed in detail. This helps identify the research gaps, and
laid the foundation of the results
presented in Chapter 5-7.
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
30
2.2.1 Use of time synchronisation in power systems
Precision time synchronisation in power systems is challenging, as
it not only requires
accurate timing across a large geographical area, but also secure
and reliable time information
distribution must be maintained at all times. A common reference
time is critically important
for grid monitoring, real-time situation awareness, coordinating
system operation and
protecting power systems and their assets. However, issues can be
found due to bad antenna
mounting, bugs in the product firmware, interoperability, and
reliability of synchronisation
architecture. Utilities, academia, independent laboratories, and
government are working on
improving the timing accuracy with advanced technologies and
achieving an interoperable
feature across commercial devices [23]. Efforts are also underway
to increase the security
level, detect the anomalous behaviours and form a best practice to
build a resilient and reliable
timing system.
Most PAC applications need timing accuracy in different levels from
sub-microsecond to
seconds. Currently, various industry standards published by the
IEEE Power Systems Relay
Committee (PSRC), the IEC Technical Committee 57 - Power Systems
Management and
Associated Information Exchange Committee specifies requirements
and provides guidance
to achieve the desired level of time synchronisation [24]–[26]. A
brief summary of key grid
timing requirements is provided as follows.
2.2.1.1 1 µs or less
A travelling wave fault locator normally synchronises two wave
detectors at each end of the
line. When a fault occurs, an electrical wave is initiated at the
point of fault and travels in
opposite directions at approximately the speed of light. Each
detector identifies the fault
instigated wave and timestamps the arrival time. Comparing the time
information from both
terminals, the fault location can be determined accordingly.
Precise timing ensures an
accurate fault location detection [27].
Synchrophasors contain high precision timestamps for voltage and
current measurements. A
time error of 1 µs would lead to 0.022° phase error at 50Hz
according to IEEE C37.118 [24].
These synchrophasors, typically with a sampling rate between 30 and
120 samples per second,
are sent from local phasor measurement units (PMUs) to the phasor
data concentrators
(PDCs). They are then compared to provide system operators with
grid conditions or to be
used for wide area monitoring, protection and control (WAMPAC). A
time synchronisation
error will result in angle shifts in phasor measurement, which
raises issues with phase angle
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
31
monitoring, and other protection and control applications such as
anti-islanding protection,
droop control and wide-area oscillation damping [28], [29].
Another application that requires precision timing in
sub-microsecond region is the
IEC 61850 process bus that uses SV measurements [30]. Current and
voltage measured by
instrument transformers are digitalised by standalone merging units
(MUs) into sampled
values with the rate of 80 samples per cycle. SV streams carry a
field called sample counter,
which is an incremental value from 0 to 3999, at 50 Hz. The counter
resets back to zero at the
start of each second. A common time reference ensures these SV
streams can be time aligned
with each other.
Line differential protection or transformer differential protection
have a slightly less strict
requirement for time synchronisation; normally an accuracy level of
10 to 20 µs is
recommended [16]. Protection operations are based on the bias
characteristics in which bias
and differential currents are compared with the restraint curve
configured in the relay. A
synchronisation deviation can cause incorrect differential current
measurement, increasing
the risk of relay maloperation [31].
2.2.1.3 1 ms or less
The synchronisation with 1 ms is required by supervisory control
and data acquisition
(SCADA) and digital fault recorders (DFRs). SCADA scans the network
every a few seconds
to have an overview of the grid conditions. Information exchange
contains breaker status,
transformer temperature, voltage and current measurement; these all
need time information
to mark when the data is obtained [23]. Similarly, DFRs timestamp
when the fault happens
and record the current and voltage waveforms before, during, and
after the fault [32].
2.2.2 Timing methods and challenges in delivering accurate
time
information
Time synchronisation is necessary for substations to establish a
coherent time frame among
various devices. It is particularly important for network devices
to have a reasonably accurate
sense of time in terms of analysing the sequence of events which
occurred in a power system
and to provide quick response to the network contingencies. IEC
61850-5 specifies various
time synchronisation classes from low (>10 ms) to the most
stringent accuracy (≤1 µs) [12]
as shown in Table 2-1. In general practice, for synchronising
different events, 1 ms accuracy
is sufficient. However, for aligning voltage and current values
from various locations, an
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
32
accuracy down to 1 µs is required. In other words, for IEC 61850
substations, synchronising
GOOSE messages based on IEC 61850-8-1 [33], 1 ms accuracy is
reasonable, Whereas, time
accuracy must meet 1 µs for synchronisation of SV, according to IEC
61850-9-2 Light
Edition (9-2 LE) [30].
Table 2-1: Time synchronisation classes specified in IEC 61850-5
[12].
Time synchronisation class Synchronisation error
TL >10 ms
T0 10 ms
T1 1 ms
T2 100 µs
T3 25 µs
T4 4 µs
T5 1 µs
There are two ways to distribute the time signals from the time
sources to various intelligent
electronic devices (IEDs) in substations, the dedicated timing
approach and the networked
timing approach. The former uses a standalone cabling system, e.g.
coaxial, twisted pair and
fibre optic cables, to covey time information. The latter transmits
the time signal via Ethernet
cabling systems shared with other substation data traffic.
2.2.2.1 Dedicated timing systems
Dissemination of time signals based on dedicated timing method can
be implemented using
either 1-PPS or IRIG-B time codes via a separated infrastructure
(Figure 2-1). 1-PPS provides
highly accurate synchronisation by transmitting one pulse every
second. The rising and
falling edge of pulses must be kept sharp with both rising and
falling time (10% to 90% and
vice-versa of the pulse level) less than 200 ns. However, 1-PPS has
a significant drawback in
that the pulses do not contain time-of-day (absolute time)
information. Hence, it must coexist
with other timing methods such as IRIG-B or Network Time Protocol
(NTP) if applications
demand absolute time [34]. 1-PPS became the recommended timing
solution in 9-2 LE for
process bus applications since MUs do not require absolute time
information for
synchronisation. In the recent published standard IEC 61869-9, the
synchronisation method
based on IEEE 1588 becomes the primary timing method for digital
interface
implementations of instrument transformers, with an option of 1-PPS
[35].
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
33
Figure 2-1: An example of dedicated timing methods used for
synchronising various IEDs
in a substation automation network [36].
The history of IRIG-B time code can be traced back to 1952 when the
U.S. guided missile
test ranges established the Inter-Range Instrumentation Group with
an aim to standardise
different time codes [37]. IRIG contains a set of time code formats
starting with one letter
followed by three digits. These designations reflect different
attributes of the corresponding
time code. The current IRIG standard 200-04 contains alphabetic
designations of A, B, D, E,
G, and H [38]. IRIG-B is the most commonly used dedicated time
synchronisation method in
substations; it is encoded as binary coded decimal in one second
window to convey absolute
time information. IRIG-B can be transmitted either as raw pulses
(B003 and B004) via fibre
optic cables or copper cables using coaxial or twisted pair, or as
amplitude modulated (B124)
with 1 kHz carrier signals over coaxial cables. Unmodulated IRIG-B
is able to achieve timing
accuracy up to sub-microsecond level. Over the years, IRIG-B has
introduced various
extensions, providing more information such as year, day light
saving time, offset to
Coordinated Universal Time (UTC), leap second information and time
quality to meet the
requirement for new applications.
However, one of the challenges faced by the substation designer is
that the client devices in
the same network generally are not supplied by the same vendor.
These devices may not be
able to accept all IRIG-B codes since they demand different
synchronisation accuracy levels.
This leads to a mixed-use of both modulated and unmodulated IRIG-B
time code with
different distribution methods (copper or fibre), complicating the
substation timing network
design significantly.
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
34
In addition, although 1-PPS and IRIG-B are well-established
techniques, they require
dedicated cables to distribute the time signals. A substation
control room is often a few
hundred meters away from outdoor cubicles in a large open-air
substation. A large number of
copper cables or optic fibres therefore need to be deployed. In
addition, both 1-PPS and IRIG-
B are strictly one way methods (time information is communicated
only from time source to
the clients), meaning they cannot automatically compensate the
propagation delay. The
manual delay compensation demands considerable effort during
installation and maintenance,
especially in large substations. As the distributed systems are
moving quickly towards using
Ethernet as a sole communication medium, the dedicated timing
methods are no longer the
preferable option and are being gradually superseded by network
based timing methods [39].
2.2.2.2 Networked timing systems
NTP and PTP, generally referred to as the networked timing system,
distribute time
information through packet-switched networks to synchronise various
devices (Figure 2-2).
Inside a substation, the transmission medium is predominately
Ethernet, whilst time domain
multiplexing (TDM) based synchronous digital hierarchy (SDH) or
synchronous optical
networking (SONET) and are used more often in wide-area networks
(WAN). To achieve a
highly accurate synchronisation over packet-switched Ethernet
networks, a two-way time
transfer approach is adopted. Periodical dedicated packet exchange
is established between a
reference node (server or grandmaster) and the rest of the nodes
(clients or slaves) in a system.
Figure 2-2: An example of networked timing methods (NTP or PTP)
used for synchronising
various IEDs in a substation automation network [36].
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
35
NTP is one of the oldest and most established communication
protocols for networked timing.
The protocol was introduced by David L. Mills from The University
of Delaware in the 1980s
[40]. NTP is widely used in WAN to synchronise computers and
servers with the highest
achievable accuracy in millisecond range and has been adopted in
recent years for substation
use [41]. NTP transmits Coordinated Universal Time (UTC) time scale
in a server-client
manner; this meets the requirement of industrial standards such as
IEC 61850 (without the
process bus) and IEEE 1815 (Distributed Network Protocol 3) that
all timestamps shall be
based on an existing time standard [12], [42].
IEEE 1588-2008, known as “PTPv2”, is the second edition of PTP
after the first version was
published in 2002. PTPv2 provides significant improvement in terms
of synchronisation
accuracy up to sub-microsecond with the aid of hardware time
stamping techniques. It forms
the timing hierarchy in a master-slave framework so only one clock
can be elected as a
grandmaster clock to synchronise all slave clocks in the network.
In contrast to NTP, PTP is
based on International Atomic Time (TAI) which solves the inherited
issue of UTC as a
discontinuous timescale. UTC uses leap seconds added or subtracted
from the current time
once every a few years to account for the variation in the rate of
rotation of the earth. This
means all devices synchronised to UTC have to act accurately at the
point of discontinuity
otherwise the timing accuracy will be violated significantly. Using
TAI as a single time
reference avoids dealing with leap second issues. The offset to UTC
is embedded into
synchronisation messages so UTC time can be easily derived at the
slave clocks.
One of the benefits of networked timing is that NTP and PTP shares
the same communication
channel with other substation automation protocols such as IEC
61850, Distributed Network
Protocol 3 (DNP3), IEC 60870-5-104, and Modbus so a great saving
can be achieved by
eliminating dedicated timing systems. Also, both protocols use
two-way clock
synchronisation method, so propagation delay can be computed and
compensated at the end
nodes.
The basic working principle of time synchronisation in a
packet-switched networks, including
NTP and PTP, is generalised in Figure 2-3. In the downstream
direction, the sending time T1
is encoded in the timing message and the recipient time of the same
message at the end node
is denoted as T2. T2-T1 equals to the transmission delay plus the
offset of the local clocks
between the source and end node as shown in (2-1). Similarly, two
timestamps T3 and T4 can
be obtained using upstream message transfer. In this case, the
upstream transmission delay
can be calculated as T4-T3 subtracting the clock offset, shown in
(2-2). These two timestamps
are eventually communicated to the end node for its local clock
adjustment.
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
36
Figure 2-3: Basic working principle of time synchronisation in a
packet-switched network
[43].
2 = 1 + delay−downstream + offset (2-1)
4 = 3 + delay−upstream − offset (2-2)
Assuming the upstream and downstream transmission delay are
identical, the end nodes can
utilise the timestamp information to derive both transmission delay
and clock offset as
indicated in (2-3) and (2-4).
delay−downstream = (2 − 1) + (4 − 3)
2 (2-3)
2 (2-4)
In the above calculations, an underlying assumption is the time it
takes for a PTP message to
transfer from the master to the slave (downstream delay) is the
same as the time for a PTP
message to transfer from the slave to the master (upstream delay).
However, in a packet-
switched network this is not always the case and, in fact, nearly
impossible with various nodes
between transmitter and receiver. Asymmetrical delay increases
inaccuracies of clock offset
calculation, which jeopardises the overall synchronisation
performance.
There are two major factors that can result in delay asymmetry. The
first is related to the
accuracy of timestamping, i.e. how precise the timestamp reflects
the moment of the message
either appearing on or arriving from the network. NTP uses a
central processing unit (CPU)
to draw timestamps; this method is called software timestamping.
When a node is sending a
CHAPTER 2: TIME SYNCHRONISED SUBSTATION AUTOMATION NETWORKS USING
IEEE 1588
37
timing message, a timestamp is taken while the CPU is assembling
the packet. After drawing
the timestamp, the packet need to be moved from memory to the
egress port buffer to be
transmitted. This process can be affected by various factors (e.g.
CPU scheduling), so the
delay is not constant. Depending on the occupancy of the egress
buffer, a further delay may
be experienced if other packets are being transmitted. Hence, the
time span between when the
timestamp is drawn and the moment the packet appears at the egress
port is not deterministic
and is difficult to predict precisely. Likewise, the receiving
packet suffers from the same
inaccuracy due to the time variation in moving the packet from the
ingress port to the point
where the timestamp is taken, normally located at higher layers
(e.g. application level)
according to the open system interconnection (OSI) model.
One solution to increase the