© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public Nanosecond Accuracy Timing Solution for Data Center Fabric BRKDCT-2215 1
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Nanosecond Accuracy Timing Solution for Data Center Fabric BRKDCT-2215
1
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Timing requirements and solution
IEEE 1588 clock type and how does it work
IEEE 1588 message details
IEEE 1588 functions and operation on Nexus switch
Nexus 3000 PTP performance benchmarking
PTP network design considerations
PTP performance validation in solution testing lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Precision Timing is Fundamental
Clock is the one of the most important component of any modern electrical system
Network and applications also need accurate timing information to correlate all the events
‒ Network diagnostics
‒ Application transactions
‒ Digital forensics
‒ Data and event-log analysis are all based on accurate timing information
Accurate timestamp is also a mandatory requirement for compliance
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Timing Challenge for Today’s Network
Financial institutes are building faster trading systems
‒ To gain competitive advantage to increase profitability
Switches can forward the packet in a matter of microsecond
‒ Cisco Nexus 3000 ultra low latency switch
HPC clusters need accurate timing synchronization solution
‒ Some HPC applications have strict timing synchronization requirement, such as performance analysis on parallel process. This requirement becomes more important today as the cluster size is getting bigger and bigger
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Timing Challenge for Today’s Network
Switches can forward the packet in a matter of microsecond
• Nexus 3000 ultra low latency switch can forward the packet (64B) in 800ns!
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 6 Cisco Confidential 6 © 2010 Cisco and/or its affiliates. All rights reserved.
Timing Challenge for Today’s Network Electronic Trading Environment
~100us
Sub 100us !
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Need an Accurate Timing Synchronization Solution
7
Have a precise local only clock is not practical ‒ It can be very expensive
‒ Sill have the risk to run out of sync
Time need to be precise and synchronized ‒ Everyone will have the same timing information
‒ Solve the problem for distributed system
‒ Should be easily distributed
‒ Should meet the accurate requirement
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Comparison of Different Timing Source
NTP (Network Time Protocol) o Traditionally NTP is used to provide timing information on packet network.
However, its accuracy usually is limited at milliseconds level
GPS (Global Position System) o GPS with proper installation and calibration can provide 100ns accuracy. However,
It requires separate network and media (not Ethernet). It’s costly, not straight forward to deploy in a large scale
IRIG (And other serial timing protocols) o Legacy protocol mostly used in environment requires millisecond accuracy. Also
facing the same challenge as GPS
PTP (Precision Timing Protocol) o Defined in IEEE1588, distributed time synchronization protocol for packet network.
Can provide nanosecond accuracy.
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 Precision Time Protocol
IEEE 1588 Precision Time Protocol (PTP) is a highly accurate distributed time synchronization protocol for packet network
IEEE 1588-2008, as known as IEEE 1588v2 or PTPv2 is the latest IEEE 1588 standard.
• Can direct map to Ethernet, or UDP IPv4.
• Packet based timing distribution and synchronization.
• Nanosecond to sub-microsecond accuracy
• Low administrative effort, easy to manage and maintain
• Low cost and low resource use, works on high-end or low-end device
• Support redundant and fault-tolerant
• No need to implement costly GPS or other dedicated timing network
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Industry Benefitted by IEEE 1588v2
Telecommunications PTP Telecom Profile ITU G.8265.1 ITU G.8275.1
Industry automation (Default profile)
Financial
Smart grid (Power profile, IEEE C37.238)
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 For Financial and DC Network
Industries
• Financial Trading • High Performance Computing • Massively Scalable Data Centers
Applications @ Server
• Application latency measurement • Event & Log correlation between systems • Measure “time to trade”
• Switch & Hop-by-Hop Latency Measurements • Accurate Timestamp of Monitored Traffic for ERSPAN
Applications @ Switch
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Timing requirement and solution
IEEE 1588 clock type and how does it work
IEEE 1588 message details
IEEE 1588 features supported on Nexus switch and configurations
Nexus 3000 IEEE 1588 performance benchmarking
IEEE 1588 PTP network design considerations
PTP performance validation in solution testing lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 Clock Type
Master Clock (MC)
• Master clock provide reference time for one or more slave clocks.
• A grandmaster (GM) is the highest-ranking clock within its PTP domain, it’s the primary reference source (PRS) for all other PTP elements
Slave Clock (SC) • Each slave clock synchronizes itself to the master clock
Ordinary Clock (OC) • Has a single PTP port in a domain and maintains the timescale used in
the domain. It could be a master clock, or a slave clock
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 Clock Type – Cont’d
Boundary Clock (BC)
• Has multiple PTP ports in a domain and maintains the timescale used in the domain. It could be a master or slave clock in the same time on different port
Transparent Clock (TC) • A device that measures the time taken for a PTP event message to
transit the device, and compensate the packet delay by updating the timestamp.
• End-to-End transparent clock
• Peer-to-Peer transparent clock
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Master Clock Selection
Based on IEEE 1588v2 BMCA
Announce messages are exchanged among potential grand masters. BMCA (Best Master Clock Algorithm) runs locally on each port. It compares its own data set with the received data set to determine the best clock based on the attributes with following priority:
1. Priority1(0-255)
2. Class (clockClass)
3. Accuracy (clockAccuracy)
4. PTP variance (offsetScaledLogVariance)
5. Priority2 (0-255)
6. Identifier (IEEE EUI-64) (clockIdentity)
The number of hops between local clock and master is also used to as a tie-breaker
The status of the port can be master, slave or passive
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Hierarchy Network Clock Topology
1 Elect the grand master, form a master-slave hierarchy. Grand master is selected based on Best Master Clock selection Algorithm (BMCA). (Master clock 1 is selected as Grand Master in the diagram)
2 Each slave clock synchronizes itself to the master clock
Master Clock 1 Master Clock 2
Boundary Clock (1) Nexus switch
Boundary Clock (2) Nexus switch
Slave Slave Slave Slave Slave Slave
Grand Master
Boundary Clock (3) Nexus switch
Server farms
Nexus Switches
Clock source
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 Clock Synchronization
Clock Offset
t2 - t1- mean path delay – Sync Message correction field
Mean Path Delay
((t2 - t1) + (t4 – t3)) / 2
After the synchronization
Slave clock derives Time of Day, phase and frequency signals from the master
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Timing requirement and solution
IEEE 1588 clock type and how does it work
IEEE 1588 message details
IEEE 1588 features supported on Nexus switch and configurations
Nexus 3000 IEEE 1588 performance benchmarking
IEEE 1588 PTP network design considerations
PTP performance validation in solution testing lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 Message Types
General messages (not time stamped) • Announce
• Follow_Up
• Delay_Resp
• Pdelay_Resp_Follow_Up
• Management and signaling
Event messages (need to be accurately time stamped) • Sync
• Delay_Req
• Pdelay_Req
• Pdelay_Resp
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 Message Types
The Announce message is used to establish the synchronization hierarchy
The Sync, Delay_Req, Follow_Up, and Delay_Resp messages are used to synchronize ordinary and boundary clocks
The Pdelay_Req, Pdelay_Resp, and Pdelay_Resp_Follow_Up messages are used to measure the link delay in transparent clocks
The signaling messages are used for communication between clocks for all other purposes
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588v2 Packet Details
Nexus switches support PTP over UDP over IPv4 multicast • Communication between master and slave use IPv4 multicast group
address 224.0.1.129
• Event messages use UDP Port 319
• General message use UDP port 320
• Above applies to both unicast and multicast
Nexus switches current doesn’t support PTP unicast communication
IANA also reserved additional multicast address for PTP, currently it’s not used
• 224.0.1.130
• 224.0.1.131
• 224.0.1.132
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Time-of-Day
IEEE 1588v2 PTP is capable of frequency, phase and time-of-day synchronization
Telecommunication industry requires the synchronization of frequency, phase and time-of-day
Most of the applications in financial institute and data center networks are interested in Time-of-Day synchronization
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Timing requirement and solution
IEEE 1588 clock type and how does it work
IEEE 1588 message details
IEEE 1588 functions on Nexus switch and configurations
Nexus 3000 IEEE 1588 performance benchmarking
IEEE 1588 PTP network design considerations
PTP performance validation in solution testing lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Feature Supported on Nexus Switches
PTP is supported on all Nexus 3000 platforms
• Nexus 3048 (10/100/1000M with 10G uplinks)
• Nexus 3064 (100M/1G/10G with 10G/40G uplinks)
• Nexus 3016 (16 x 40G with flexible 10G/40G template)
PTP is supported on all F series line cards in Nexus 7000 platforms
PTP is supported on all Nexus 5500 platforms
Interface supported by PTP function
• All L2 / L3 physical interface
• Port-Channel interface (including VPC interface)
• Not supported on sub-interface and SVI
Nexus platforms support 2-step boundary clock
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IEEE 1588 Implementation in Nexus 3000
1. 1588 packet is time-stamped at ingress of ASIC to record the t2,
2. Timestamp points to the first bit of the packet (following SFD)
3. Packet is copied to CPU with timestamp and destination port
4. The packet is going through PTP stack and other process
5. The packet is sent to egress port.
6. The corresponding timestamp for the TX packet is available from the FIFO TX time stamp
PCI-Local Bus
SPI eUSB Flash
USB Conn
Console
Management Ethernet
48 x XFI XLAUI/XFI
48 x SFP + Cages 4 x QSFP
Switching ASIC
PTP Stack, Clock Servo,
Frequency Synchronizer
4GB DRAM
CPU
1 2
3 4
5
Nexus 3000 Data Center Switch
6
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Configuration on Nexus Switches
Required steps:
Enabling PTP feature (no license required)
Configuring an IP address for PTP communication, one for each PTP node
Enabling PTP on corresponding interfaces
n3k(config)# feature ptp n3k(config)# interface loopback0 N3k(config-if)#ip address 1.2.3.4/32 n3k(config-if)# ptp source 1.2.3.4 n3k(config-if)# interface e1/1 n3k(config-if)# ptp
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Configuration on Nexus Switches
Optional steps:
Configuring priority values, it’s a global value, lower number has higher priority
Configure PTP packet rate on interface
Use PTP to update system calendar
Currently Nexus 3000 requires 4 PPS or higher sync packet rate from neighboring master
n3k(config)# ptp priority 1 <priority 1> n3k(config)# ptp priority 2 <priority 2> n3k(config)# interface e1/1 N3k(config-if)#ptp sync interval ? <-6-1> Log seconds
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Verify PTP Operation on Nexus Switches
Verify PTP interface status Verify PTP clock status
n3048-1# sh ptp brief PTP port status ----------------------- Port State ------- -------------- Eth1/16 Master Eth1/12 Master Eth1/11 Slave Eth1/9 Master Eth1/6 Disabled Eth1/4 Master Eth1/3 Master n3048-1#
n3k-p1# sh ptp clock PTP Device Type: Boundary clock Clock Identity : 0: 5:73:ff:ff:ee:cb:41 Clock Domain: 0 Number of PTP ports: 4 Priority1 : 255 Priority2 : 255 Clock Quality: Class : 248 Accuracy : 254 Offset (log variance) : 65535 Offset From Master : 528 Mean Path Delay : 412 Steps removed : 1 Local clock time:Fri Apr 13 06:51:48 2012 n3k-p1#
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Verify PTP Operation on Nexus Switches
Verify PTP master clock Verify PTP clock status
n3k-p1# sh ptp parent PTP PARENT PROPERTIES Parent Clock: Parent Clock Identity: 0:b0:ae:ff:fe: 2:a8:9b Parent Port Number: 1 Observed Parent Offset (log variance): N/A Observed Parent Clock Phase Change Rate: N/A Grandmaster Clock: Grandmaster Clock Identity: 0:b0:ae:ff:fe: 2:a8:9b Grandmaster Clock Quality: Class: 6 Accuracy: 33 Offset (log variance): 25600 Priority1: 128 Priority2: 128
n3k-p1# sh ptp time-property PTP CLOCK TIME PROPERTY: Current UTC Offset valid: 1 Current UTC Offset: 34 Leap59: 0 Leap61: 0 Time Traceable: 1 Frequency Traceable: 1 PTP Timescale: 1 Time Source: 0x20(GPS) n3k-p1#
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PONG - Pulse of the Network
switch# configure terminal switch(config)# pong destination-swid 3506 destination-mac 001b.54c2.9a43 vlan 1 count 3 Packet No. 1 --- ----------------- -------------------------- Hop Switch-id Switching time (sec, nsec) --- ----------------- -------------------------- 1 0-1b-54-c2-9a-41 0 4752 2 0-1b-54-c2-9a-43 0 544258088 3 0-1b-54-c2-9a-41 0 4792 Round trip time: 0sec 15416 nsec Packet No. 2 --- ----------------- -------------------------- Hop Switch-id Switching time (sec, nsec) --- ----------------- -------------------------- 1 0-1b-54-c2-9a-41 0 4752 2 0-1b-54-c2-9a-43 0 522744240 3 0-1b-54-c2-9a-41 0 4736 Round trip time: 0sec 15368 nsec Packet No. 3 --- ----------------- -------------------------- Hop Switch-id Switching time (sec, nsec) --- ----------------- -------------------------- 1 0-1b-54-c2-9a-41 0 4752 2 0-1b-54-c2-9a-43 0 521869920 3 0-1b-54-c2-9a-41 0 4800 Round trip time: 0sec 15360 nsec
PONG measures the switch port to port delay by utilizing the underlying PTP infrastructure
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Troubleshoot - SPAN
Use Switched Port Analyzer (SPAN) to monitor PTP packet
Use Wireshark to decode the captured packets
SPAN configuration:
monitor session 1 source interface Ethernet1/11 both destination interface Ethernet1/8
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Troubleshoot - Enanalyzer
n3048-1# ethanalyzer local interface inbound-hi detail display-filter ptp limit-captured-frames 1 Capturing on eth4 … snap … Ethernet II, Src: 54:7f:ee:14:6d:fc (54:7f:ee:14:6d:fc), Dst: 01:00:5e:00:01:81 (01:00:5e:00:01:81) Destination: 01:00:5e:00:01:81 (01:00:5e:00:01:81) Address: 01:00:5e:00:01:81 (01:00:5e:00:01:81) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: 54:7f:ee:14:6d:fc (54:7f:ee:14:6d:fc) Address: 54:7f:ee:14:6d:fc (54:7f:ee:14:6d:fc) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol, Src: 11.11.55.1 (11.11.55.1), Dst: 224.0.1.129 (224.0.1.129) Version: 4 Header length: 20 bytes Differentiated Services Field: 0xd4 (DSCP 0x35: Unknown DSCP; ECN: 0x00) 1101 01.. = Differentiated Services Codepoint: Unknown (0x35) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length: 82 Identification: 0x0000 (0) Flags: 0x00 0... = Reserved bit: Not set .0.. = Don't fragment: Not set ..0. = More fragments: Not set
Ethernet Header
IP Header
CLI: ethanalyzer local interface inbound-hi
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Troubleshoot - Enanalyzer [Continued] Fragment offset: 0 Time to live: 64 Protocol: UDP (0x11) Header checksum: 0x944d [correct] [Good: True] [Bad : False] Source: 2.2.2.1 (2.2.2.1) Destination: 224.0.1.129 (224.0.1.129) User Datagram Protocol, Src Port: 319 (319), Dst Port: 319 (319) Source port: 319 (319) Destination port: 319 (319) Length: 52 Checksum: 0x45e7 [correct] [Good Checksum: True] [Bad Checksum: False] Precision Time Protocol (IEEE1588) versionPTP: 2 versionNetwork: 44 subdomain: messageType: Unknown (0) sourceCommunicationTechnology: PROFIBUS (5) sourceUuid: 73:ff:ff:ee:cb:41 (73:ff:ff:ee:cb:41) sourcePortId: 8 sequenceId: 36660 control: Sync Message (0) …… snap …..
UDP Header
PTP Header
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Troubleshoot - CoPP
PTP CoPP policer is set to 1000 pps (packet per second) by default
NX-OS provides configurable CoPP to allow user to tune the CoPP parameters to meet their particular need.
n3k-p1# sh policy-map interface control-plane | section ptp class-map copp-s-ptp (match-any) police pps 1000 OutPackets 100054646 DropPackets 0 n3k-p1# n3k-p1(config)# policy-map type control-plane copp-system-policy n3k-p1(config-pmap)# class copp-s-ptp n3k-p1(config-pmap-c)# police pps ? <0-20000> Packet per second value n3k-p1(config-pmap-c)# police pps
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Timing requirement and solution
IEEE 1588 clock type and how does it work
IEEE 1588 message details
IEEE 1588 functions and operation on Nexus switch
Nexus 3000 PTP performance benchmarking
PTP network design considerations
PTP performance validation in solution testing lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance Validation
Two important metrics are used to verify the PTP performance* • Accuracy: Clock (Time of the Day) offset between GMC and slaves
• PDV (Packet Delay Variation)
Accurate Time-of-day is what the financial trading environment and data center applications are looking for
Factors need to be considered during performance validation • Traffic load and patterns
• Network topology, number of hops
• Type of PTP grand master, slaves
• PTP protocol parameters (packet rate, etc *Currently there is no standard on how to benchmark PTP BC or TC devices
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance Validation - HW client
PTP Grandmaster
Oscilloscope
PTP HW client (Cisco 2941)*
1PPS 1PPS
PTP Protocol over 1GbE Ethernet
*Cisco 2941 is used because it’s the only available PTP hardware client with GE interface when the test was performed
22.9ns
GPS satellite
GPS Antenna
Step 1: Baseline
Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Std Dev (ns)
4.5 hours 22.9 -3.5 47.5 14.2
Counter
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance Validation – HW client
PTP Grandmaster
Oscilloscope
PTP HW client (Cisco 2941)*
1PPS 1PPS
PTP Protocol over 1GbE Ethernet
265 ns
GPS satellite
GPS Antenna
Step 2: Add 1 Nexus 3064
Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Std Dev (ns)
12 hours 265.29 82 418.5 76.19
Counter
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance Validation – HW client
PTP Grandmaster
Oscilloscope
PTP HW client (Cisco 2941)*
1PPS 1PPS
389 ns
GPS satellite
GPS Antenna
Step 3: Add 2 Nexus 3064s
Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Std Dev (ns)
12 hours 389.2 117.5 569.8 92.4
Counter
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance Validation – HW client
PTP Grandmaster
Oscilloscope
PTP HW client (Cisco 2941)*
1PPS 1PPS
391 ns
GPS satellite
GPS Antenna
Step 4: Congest the network link
Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Stdev (ns)
4.5 hours 391.5 117.5 569.8 84.6
Counter
Congested Traffic tool
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance Validation – HW client
PTP Grandmaster
Oscilloscope
PTP HW client (Cisco 2941)*
1PPS 1PPS
-1.05 ns with large variance
GPS satellite
GPS Antenna
Step 5: Stop the traffic, disable PTP on Nexus switches
Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Stdev (ns)
12 hours -1.05 3743 4985 126.7
Counter
PTP function disabled
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance – PDV Validation PDV (Packet Delay Variation) could be introduced by queueing delays,
data path changes, congestion, etc.
PDV can reduce the accuracy of the network timing protocol
Sync PDV and Delay_Req PDV are the two most significant PDVs
PDV can be accumulated in a chain of PTP clocks
Boundary clock can help reduce the PDV accumulation in the network
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance – PDV Validation
PTP Grandmaster
GPS satellite
GPS Antenna
PTP is enabled
PDV generator and measurement tool
PDV should be measured under different traffic load
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Performance – PDV Validation
80% traffic load PDV=~400
Link congested PDV=~400
One-hop 7 hour PDV test
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP vs. NTP Performance Comparison
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP vs. NTP Performance Comparison
Average offset: NTP: 263us PTP: 0.412us
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Components of End-to-End Solution
Grandmaster Clock • GPS (or equivalent) clock input
• What’s the oscillator options
• What and how many interfaces it support
• What’s the protocol it supported
• What kind of I/O output it provide
• What’s the management capability
Oscillator Interface Protocol I/O output Management
Rubidium 10GbE* Unicast/Multicast 1PPS Web GUI
OCXO 1GbE 1-step / 2-step 10Mhz SSH / SNMP
TCXO 100M NTP GbE (fibber or copper) Authentication
*Not available on current market
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Components of End-to-End Solution
PTP Clients • Hardware-assistant or software only
• Oscillator options (hardware-assistant)
• What I/O interfaces it supports
• What’s the protocol it supported
• Fault-tolerant capability
• Management capability
Oscillator * I/O Options * OS requirement Protocol Fault-Tolerant
OCXO 1PPS in/out Windows Unicast/Multicast Switch-over
TCXO Other time code Linux/Unix 1-step / 2-step Fail back to NTP
Ethernet 32 / 64 bits NTP Never go back!
* Applicable to hardware-assistant client only
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Timing requirement and solution
IEEE 1588 clock type and how does it work
IEEE 1588 message details
IEEE 1588 functions and operation on Nexus switch
Nexus 3000 PTP performance benchmarking
PTP network design considerations
PTP performance validation in solution testing lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Network Design Considerations
Site survey & requirement
gathering
Define benchmarking methodology
Define network
topology and solution
component
POC testing and/or pilot
run
In production, continue
monitoring the network
It’s time to build an End-to-End PTP network!
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Network Design Considerations
Site survey & requirement gathering ‒ application requirement
‒ server requirement
‒ network load, traffic pattern, etc
Benchmarking methodology: PDV or offset? ‒ application requirement
Where is the grand master?
How many layer does network have?
What’s the scalability requirement?
Does it require dedicated PTP switch?
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Network Design Considerations In-band Solution
• Install & configure precision
timing source (e.g., GPS and grand master)
• Enable PTP on all Nexus switches in the existing data network
• Install & configure PTP client on existing servers
• For greenfield implementation
Core Layer
Server Farms
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
• Install & configure precision
timing source (e.g., GPS and grand master)
• Install & configure dedicated
PTP switches (boundary clock) as a PTP distribution switch
• Install & configure PTP client
on existing servers
• For retrofit implementation
Core Layer
PTP Network Design Considerations Out-of-band Solution
Server Farms
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Network Design Considerations PTP as a Service From Exchange or DC IT
• PTP can be provided by the
Exchange as precision time synchronization service to its customers
• Data center service team can also provide PTP as a service to its client and applications.
• Can be done via in-band, out-
of-band or hybrid mode
Backbone
Customer A Customer B Customer C
Customers
Provider Site A Site B
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Timing requirement and solution
IEEE 1588 clock type and how does it work
IEEE 1588 message details
IEEE 1588 functions and operation on Nexus switch
Nexus 3000 PTP performance benchmarking
PTP network design considerations
PTP performance validation in solution testing lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Validated in Cisco Customer Solution Lab
Precision Time Source – Grandmaster with GPS receiver
Dedicated PTP distribution switches – 1st hop, 2 x nexus 3048s
Spine/Aggregation layer – 2nd hop, all Nexus 3064s
Leaf/Access layer 3rd hop, all nexus 3064s, each N3064 has 20-30 servers
In-band Solution
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PTP Test Results in Cisco Solution Lab
57
Server location PTP Client Nexus Platform Accuracy (usec)
1st Hop Cisco PTP Client (v3.8) Nexus 3048 pos avg 0.351768496 neg avg -1.541941669
2nd Hop Cisco PTP Client (v3.8) Nexus 3064-E pos avg 0.511412221 neg avg -1.487674687
3rd Hop Cisco PTP Client (v3.8) Nexus 3064-E pos avg 1.269494697 neg avg -1.299382558
Open Source PTP client (OS-2.2.1) is also tested. Its performance results are similar but experienced failover problems.
NX-OS Server Spec Test Duration Sync packet rate (PPS)
5.0(3)U2(2) UCS 200-M1(1x Intel E5504 processor)
Redhat Enterprise 5.4 12 hours
4 PPS (packet per second) between Nexus switches and
GM, 1 PPS towards server
Configurations Test bed is fully loaded with unicast/muticast traffic. All major L2/L3, multicast, QoS features are configured to simulate a customer production network
Performance test results
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Additional Sessions
BRKSPG-2170 ‒ Synchronization in Packet-based Networks (SyncE & IEEE1588-2008)
BRKDCT-2214 ‒ Ultra Low Latency Data Center Design
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Looking forward
ITU standard update (lots of work need to be done )
Security will be a concerns when PTP deployment become mainstream
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Key Take Away
We need a highly accurate timing synchronization solution in sub-microsecond level
IEEE 1588 PTP is a widely used timing synchronization protocol in packet network
Cisco Nexus data center switches support PTP in hardware today
Cisco Nexus switch can delivery accurate PTP timing information to client under heavy network load
PTP solution need to be carefully designed and reviewed before enabled in production network
Large-scale in-band PTP deployment is verified in Cisco solution lab
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Complete Your Online Session Evaluation Give us your feedback and you
could win fabulous prizes. Winners announced daily.
Receive 20 Passport points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.
61