When every nanosecond counts White Rabbit training · White Rabbit technology: Time reference Atomic clocks count time relying on the resonance frequency of atoms excited by microwave,

When every nanosecond counts

www.sevensols.com

White Rabbit training

Seven Solutions

White Rabbit technology


White Rabbit (WR) is a technology born at CERN which achieves sub-nanosecond

accuracy in Ethernet based networks. It allows easy deployments of scalable and

reliable networks with high accuracy synchronization requirements.


• +10 years of expertise synchronizing large scientific facilities:

– CERN, GSI, Fermilab, …

• Validated by National Metrology Institutes: NIST, NPL, PTB,

OP, VSL, ROA, VTT, RISE, …

• New PTP High Accuracy profile to be released is based on the

pre-standard approach White Rabbit.


Deterministic and highly accurate This allows saving engineering and equipment costs to achieve a global target time budget.

Cost-effective It compensates dynamically link asymmetries and temperature changes. Easy to deploy, pre-calibration.

Scalable to long distances & high numberof nodes. It supports tree topologies and daisy-chain configurations

DependableIt reduces vulnerabilities to spoofing or GPS jamming. Up to 100 km links without on site calibration.

Facilitates new services Positioning, High Frequency Trading, Time as a Service

Easy to integrate within existing infrastructures (Ethernet, PTPv2).

White Rabbit technology: Syntonization

▪ To reach subnanosecond synchronization, distributing the same clock through thenetwork is needed.

▪ Syntonization: local clock tuning based on a measure of the error between two clocks. InWhite Rabbit, the external clock and the internal reference are compared.

125.00 MHz

124.98 MHZ

125.03 MHz

Free running clocks with the same nominal frequency are not accurate enough for accurate time transfer

Not SyntonizedSyntonizingSyntonized!

L1 syntonization is used as a customized version of Sync-E to transmit the clock over the optical links. It useslocal VCXO to syntonize the local clock to the recovered clock from the link (Slave role).

White Rabbit requires specific clock circuitry to perform the syntonization and ensure accuratetime synchronization.

125.00 MHz

125.00 MHZ

125.00 MHz

White Rabbit technology: Syntonization

White Rabbit measures the offset between devices, taking into account the link asymmetry anddynamic variations because of weather conditions using picosecond level accurate timestamps.

WR slave

WR master

Timestamp generation based on frequencymixing techniques: Clock phasemeasurements

Synchronization: L1 syntonization & PTP (IEEE-1588v2)

The timestamps are capable of measuring time differences between two digital clock signals with very fine resolution(picosecond) and they are used to adjust the received and generated clock offsets.

White Rabbit technology: Ultra-accurate timestamp

White Rabbit technology: Synchronization

White Rabbit uses the information collected by the exchange of timestamped packets for correcting theconstant offset between nodes (bA≠bB)

The information from the calibration is also important for compensating the static offset between nodes.

White Rabbit technology: Topology

PPS Slaves

Time Reference

10 MHz & PPS

WR-ZEN TP WR-ZEN TPWR-SWITCH

WR

PTP Slave

PTP

PTP Switch

PTP

PTP

WR-SLAVES

WR

PPS

WR-ZEN TP-32BNC

X ns from ref.± 50 ns from ref.

< 1 ns from ref.

PTP Slaves

± 100 ns from ref.

WR-SWITCH

White Rabbit technology: Time reference

Atomic clocks count time relying on the resonance frequency of atoms excited by microwave, optical or ultravioletradiation.

• They are one of the most stable time references available.

• The most widespread atomic clocks are based on cesium or rubidium.

Any GPS receiver gets an accurate time, traceable to the atomic clocks present inside GPS satellites, as it is necessary tocalculate the receiver position.

• GPS-synchronized time receivers just ignore the position information and adjust a local tunable oscillator accordingto the data obtained from the satellites.

White Rabbit technology: Network

• The use of fiber makes encoding easier, as the optical medium does not sufferfrom crosstalk or electric interference.

• 1.25 Gbps serial datastream comes out of the serializer.

White Rabbit technology: SyncE

• In a regular Ethernet network, every node uses its own free running oscillator. Small differences of frequency between tx and rx circuits are compensated by asynchronous packetbuffers.

• Sync-E defines a hierarchycal structure where the master at the top is connected to a primaryclock.

• STM syntonizes its oscillator to the primary clock and uses this frequency to encode the data.• At the receiver end, the same frequency is recovered using PLLs and is used with lower nodes

in the hierarchy and back to the master.

White Rabbit technology: PTP

• PTP synchronizes the slave clock with a master clock

• Link delay measured with timestamped frames• Supports HW timestamps

White Rabbit technology: Phase measurements

• Ethernet packets can be received at a device at any moment.• The 125 MHz transmission clock only provides 8 ns granularity.• Phase offset measurements improve the accuracy of PTP timestamp

exchanges.

White Rabbit technology: Assimetry compensation

White Rabbit calculates link assimetries using pre-calibration:• Internal delays• SFP delays• Fiber assimetry

Δ = Δtxm + Δrxm + Δtxs + Δrxs

delayMM = Δ + δms + δsm

White Rabbit technology: Timing daemon

Link up

Syntonize

Calibrate PHYs

Measure coarse delay

Measure phase

Extend timestamps and obtain fine delay

Determine link asymmetry

Compute one-way delays and clock offset

Initial offset correction

Measure phase

Compensate for delay changes


Once syntonized, we still have to synchronize both nodes. This can be divided into two tasks:

• Coarse delay measurement, based on a PTP exchange.• The coarse delay is measured using a PTPv2 two-step packet exchange.

• Precise delay measurement that combines the coarse delay with a DDMTD phase measurement

The packets timestamps are hardware-generated:

• This measurement produces timestamp values t1, t2, t3, t4. The precision of the timestamps is extended toinclude the phase measurements: t2p, t4p.

• Due to the possibility of jitter-related problems, t2 and t4 timestamps are generated for both rising and falling edge.

• Precise round-trip delay can be calculated as delayMM = ( t4p – t1 ) – ( t3 – t2p )


• The asymmetry cannot be directly measured. It can only be estimated from delayMM and knowledge of the mediumand the transmission circuits.

• All these asymmetry sources are taken into account:• Propagation delays of eletronic components and PCB traces• Optical transceivers delay asymmetry• Fiber Rx/Tx different diffraction index.• Internals of the chips structure.

• Fiber asymmetry can be variable depending on operating conditions and link length. All the rest are consideredconstant per device.


• dTX and dRX fixed delays inside the hardware up to the optical port

• δMS and δSM are the one-way fiber delays from master to slave and back


• BiDi SFP (1310/1490 nm)

• Single-mode fiber G652

• Relative Delay Coefficient of the OF (α) • Describes the asymmetry of the fiber• The speed of propagation in the fiber depends on the wavelength

• BiDi SFPs uses λtx and λrx→Slight difference in propagation of speed

α =𝛿𝑀𝑆𝛿𝑆𝑀

− 1


Δ = Δtxm + Δrxm + Δtxs + Δrxs

delayMM = Δ + δms + δsm

α = (δms / δsm) - 1

δms= (1+α)(delayMM-Δ) / (2+α)

delayms= (1+α)(delayMM-Δ) / (2+α) + Δtxm + Δrxs)

offsetms = t1 – t2p - delayms

delayMM = ( t4p – t1 ) – ( t3 – t2p )

White Rabbit technology: Links

• White Rabbit users

https://www.ohwr.org/project/white-rabbit/wikis/WRUsers

• White Rabbit documentation

https://www.ohwr.org/project/white-rabbit/wikis/home

https://www.ohwr.org/project/white-rabbit/wikis/WRUsers

https://www.ohwr.org/project/white-rabbit/wikis/home


www.sevensols.com

Overview of products

White Rabbit technology: WR-Switch

It is the main element of the White Rabbit Technology. It has 18 SFP 1GbE ports in a 1U form factor that can be configured to work as master or slave to deploy an operational White Rabbit network.

../../Marketing/Brochures/PDF/WR SWITCH.pdf

White Rabbit technology: WR-LEN

It is the standalone WR device capable of supporting daisy chain configurations. It allows a cost-effective solution to distribute PPS/10MHz signals or IRIG-B protocol to your equipment.

../../Marketing/Brochures/PDF/WR LEN SOLUTION .pdf

White Rabbit technology: WR-ZEN

It is the interoperability element of the White Rabbit Technology. It has 2 SFP 1GbE ports that can be configured to work as master or slave to deploy daisy-chain configurations or to provide a redundant White Rabbit connection. The WR-ZEN TP devices have a redundant dual power supply. This device can obtain its time reference from an external source using both 10MHz and 1PPS signals, from another White Rabbit device through its optical ports or can work as a free-running device. It can also configure a NTP connection to a server to get the time information on its management ports.

White Rabbit technology: WR-ZEN

../../Marketing/Brochures/PDF/WR ZEN TP.pdf

../../Marketing/Brochures/PDF/WR ZEN TP FL.pdf

../../Marketing/Brochures/PDF/WR ZEN TP 32 BNC.pdf


www.sevensols.com

White Rabbit synchronization market use cases

5G Telecom Networks

• Synchronization of RRHs and BBUs

• Cross validation of time references in telecom core networks

(monitoring)

• WR over WDM

5G synchronization flow

- Centralizes Baseband Units (BBU) in a server

- BBUs all co-located, simplifying synchronization for eICIC and CoMP

Cross validation of time references in telecom core networks (monitoring)

• Synchronizing (cross-validating) Primary Time References over the core

network

Site 2

Site 3

Site 1

Core NetworkGM

WR-ZEN TP

GM

WR-ZEN TP

GM

WR-ZEN TP

WR over WDM

Fron

thau

l/backh

aul

device

sFr

on

thau

l/b

ackh

aul

dev

ice

s

………..………..………..

………..………..………..

………..………..………..

………..………..………..

Visibility network

• Distribute timing

from the time

reference to the

entire datacenter

• Deliver accurate and

precise PPS to

every capturing

device

• All capturing devices

synchronized

10 MHz & PPSWRPPS

Cabinet 3Cabinet 2Cabinet 1

WR-SWITCH

Time Reference

WR-ZEN TP-32BNC WR-ZEN TP-32BNC WR-ZEN TP-32BNC

Capturing device

Capturing device

Capturing device

Capturing device

Capturing device

Capturing device

Capturing device

Capturing device

Capturing device

Visibility network

• Tap every link

• Timestamp every packet

• A detailed live map of

the network can be

obtained for analysis

• Validated in the finance

sector by the German

stock exchange

(Deutsche Börse press

note).

Level 1

Level 2

Level 3

Level 4

GMGPS

WR-Switch

Timestamper

Timestamper

Timestamper

Timestamper

WR10 MHz

1 PPS

WR-ZEN TP-32BNC

1PPS

http://sevensols.com/index.php/2017/10/20/white-rabbit-deutsche-borse/

Jamming and spoofing protection

• A single local GNSS antenna represents an important risk.

• White Rabbit can be used to diversify the time reception.

• Deploy two or more GNSS antennas and interconnect them through a WR link.

• Jamming/spoofing a two antennas in different locations is much harder.

• Possibility to connect to a National Metrology Institute for UTC reference

alternatively to GNSS.

UTC referenceWR-ZEN TPWR-ZEN TP

10MHz

PPSUTC referenceWR

10MHz

PPS Time referenceTime reference

Saving time budget on inter metro links

• Send a local reference to a different metro location with sub-nanosecond accuracy and

precision.

• A 10 MHz & PPS or PTP reference must be provided to a WR device in location A.

• Possibility to interconnect to a National Metrology Institute for an alternate source of UTC.

• Up to 120 Km in one hop. Several hops maintaining sub-nanosecond accuracy.

• The timing can be recovered in multiple formats:

– 10 MHz & PPS

– IEE 1588

– ToD

– IRIG-B

Location A Location B

WR

Cabinet 2Cabinet 1

WR-ZEN TP NIC

WR-ZEN TP NIC

WR-ZEN TP NIC

WR-ZEN TP NIC

NIC

WR-SWITCH

Time Reference

WR-ZEN TP-32BNC

NIC NIC NIC NIC NIC

NIC NIC NIC NIC NIC NIC

Server synchronization datacenter wide

• Distribute timing from

the time reference to

the entire datacenter

• All WR network with

sub-nanosecond

synchronization

• Only one final

PPS/PTP hop

• Save PTP link

calibration

10 MHz & PPSWRPTPPPS

Inter-datacenter synchronization solutions

Timing master datacenter

Time Transfer using optical network (<80 Km)

>120 Km time Transfer using GNSSreceivers as primary time reference(APTS: network as backup)

>120 Km time Transfer using fiber asprimary time reference (GNSS forasymmetry calibration and backup)

Inter-datacenter

• Transfer of a time reference for distances up to 120 Km

• Performance is dependent of the fiber in place:

– Dedicated fiber: sub-nanosecond (< 1ns)

– DWDM fiber:

• Accuracy depending on the system: Measurable and calibrated

• Sub-nanosecond precision (< 1ns)

Inter-datacenter. Redundant timing

< 120 Km

Datacenter A Datacenter B< 120 Km

Time Reference

10 MHz & PPS

WR-ZEN TP

WR

Time Reference

WR-ZEN TP

WR-SWITCH

WR

WR

10 MHz and/or PPS

Datacenter A Datacenter B

Time distrib.to the cabinetsWR-SWITCH

Time distrib.to the cabinets

Resilient timing to GNSS jamming

Switch over in case of failure of local GNSS

Scientific facilities

• High Energy Physics. Particle accelerators

• Telescopes and sensor arrays

WR Science use case.pdf


• Synchronizing distributed instrumentation over a scientific

infrastructure

– Particle accelerators

– Distributed arrays of radio-astronomy telescopes

– Distributed sensor networks


High Energy Physics. Particle accelerators.

▪ CERN (Switzerland)

▪ GSI (Germany)

▪ LHAASO (China)

▪ Sirius (Brazil)

▪ IFMIF-EVEDA / ENS

Telescope and sensors array

▪ KM3Net (Underwater Neutrinos detector)

▪ CTA (Cherenkov Telescope Array)

▪ SKA (Square Kilometer Array)

Smart grid

• Power distribution over smart grid: Synchronizing core elements in the

Smart Grid as Phase Measurement Units (PMUs).

– Time stamping all registered data over smart grid for better detection of failures

in a forensic analysis after a black out.

Defense, Aerospace, Broadcasting…

• Ground base satellite control synchronization.

• Distributed radars needs accurate synchronization between

antennas.

• Broadcast equipment support 10 MHz and 1PPS inputs for

synchronization.

Intra-datacenter time transfer and synchronization

Intra-datacenter

PPS Slaves

Time Reference

10 MHz & PPS

WR-ZEN TP WR-ZEN TPWR-SWITCH

WR

PTP Slave

PTP

PTP Switch

PTP

PTP

WR-SLAVES

WR

PPS

WR-ZEN TP-32BNC

X ns from ref.± 50 ns from ref.

< 1 ns from ref.

PTP Slaves

± 100 ns from ref.

WR-SWITCH

Intra-datacenter

• Sub-nanosecond distribution to each cabinet

• Expend < 1ns of time error budget to each cabinet

• Choice of interoperability for last hop:

– 1 last PTP hop (±50 ns from the reference)

– 2 last PTP hops (±100 ns from the reference)

– 1 PPS hop (sync accuracy depending on clock performance of the slave)

– WR until the end node (< 1ns from the reference)

• Vendors tested: Metamako, SolarFlare, Napatech, Endace, Meinberg, …

Intra-datacenter Time Transfer and Synchronization – 1 and 2 PTP Hops

Intra-datacenter time distribution

▪ PTP interoperability using WR-ZEN TP devices.

▪ Setup 1: Direct connection to PTP Slave (± 50 ns accuracy)

▪ Setup 2: Multiple PTP slave connection using a PTP

switch (± 100 ns accuracy)

▪ Tested devices: Solarflare, Metamako, Oregano,

Endace, Napatech.

Solarflare (SF7322) Test

-25

-20

-15

-10

-5

0

5

10

15

20

25

0 300 600 900 1200 1500 1800 2100 2400 2700 3000 3300 3600

Off

set

(ns)

Samples

PPS Offset between SolarFlare devices


Arista Test:

Reference

10 MHz

1PPS

WR PTP PTP

WR switch

WR ZEN TP Arista 7150S

SF7322

100 ns


Intra-datacenter

Intra-datacenter PPS Distribution:

▪ Setup 3: Up to 32 PPS slaves connection

or 16 PPS & 10 MHz slaves connection.

▪ Tested devices: Solarflare, Metamako, Meinberg…

Intra-datacenter

MetaMako tests (PPS sync)

WR-LEN

Reference

PPS

10 MHz

WR-ZEN TP-32BNC

Fiber PPS

Metamux 32

20 ns

Inter-datacenter time transfer and synchronization

Key ideas

• PTPv2 does not provide deterministic time transfer → WR does,

– It guarantees high precision PPS distribution through all the network.

– Temperature compensated. Scenarios. Second order drift (<100ps) can be modelled and adjust in most applications if required.

• The asymmetry problem: offset on the PPS signals → Different wavelengths (different group velocities), differences on optical fibers lengths, asymmetries on the amplifiers / DCMs / DWDMs / transponders, etc...

– The asymmetries can be measured and compensated on most circumstances.

– For blind situations, a calibrated GNSS receiver is a simple and cost effective solution.

– Alternatively, we can modify the network adding bidirectional amplifiers (costlyR&D approach) or characterize all devices on the network (not always a realistic approach).


Timing master datacenter

Time Transfer using optical network (<100 Km)

>120 Km time Transfer using GNSSreceivers as primary time reference(APTS: network as backup)

>120 Km time Transfer using fiber asprimary time reference (GNSS forasymmetry calibration and backup)


• Sub-nanosecond

synchronization using

bidirectional optical fiber

links.

• Distances up to 100 Km.

• Dedicated or DWDM

optical networks channelswithout amplification.

• Dedicated or data-shared

timing channels.


< 120 Km

Datacenter A Datacenter B< 120 Km

Time Reference

10 MHz & PPS

WR-ZEN TP

WR

Time Reference

WR-ZEN TP

WR-SWITCH

WR

WR

10 MHz and/or PPS

Datacenter A Datacenter B

Time distrib.to the cabinetsWR-SWITCH

Time distrib.to the cabinets

Inter-datacenter time transfer and synchronizationTim

ing m

aster

de

viceTi

min

g m

aste

r

dev

ice

• Support for unidirectional WDM networks.

• Asymmetry calibration is provided based on GNSS receivers.


WR-ZEN TP WR-ZEN TPWR(20 Km)

Keysight 53230Frequency Counter/Timer

350 MHz – 20 ps resolution

PPS PPS

• Lab test• 24 hours measure• Sub-nanosecond accuracy and precision


Metro links

GPS-DO

PPS

10 MHz

20 Km

20 Km

20 Km

Counter20 ps

resolution

PPS

PPS

HA profileCompatible Device




16 hours

Metro links

Metro links

18 hours

GPS-DO

PPS

10 MHz

50 Km

70 Km

Counter20 ps

resolution

PPS

PPS




Metro links

-100

-75

-50

-25

0

25

50

75

100

0 3000 6000 9000 12000 15000 18000 21000 24000 27000 30000 33000 36000 39000 42000 45000 48000 51000 54000

Tim

e er

ror

(ps)

Seconds

Time Error on 120 Km, 2 hops

GNSS backup bi-fiber

• Unidirectional WDM connection through 50 km fiber

GNSS backup bi-fiber

23,06

23,08

23,10

23,12

23,14

23,16

23,18

23,20

23,22

23,24

23,26

0 2000 4000 6000 8000 10000 12000 14000

Tim

e er

ror

(ns)

Time (s)

Time error on 50 km long DWDM link

Results provided by the University of Granada with 7Sols equipment.Further results Will be presented soon.

Long Haul WR Calibration Assisted by GPS

▪ In-situ calibration impossible due to the great separation between WR nodes.

▪ GNSS techniques can be used to easily compare Master-Slave offsets

▪ It provides a check to ensure the correct functioning of terrestrial networks

▪ Suitable for network with passive or active optical equipment (fixed asymmetry)

> 40 Km

Long Haul WR Calibration Fiber Swap Method

▪ In-situ calibration impossible due to the great separation between WR nodes

▪ Different lengths in the paths

▪ Different wavelength dispersion

“Long range time transfer using optical fiber links and cross comparison with satellite based methods” Namneet Kaur

Long Haul WR Calibration Fiber Swap Method

▪ Both fibers transmitted the same wavelength

▪ Any deviation is caused only by the fiber

▪ Slave PPS is compared with a stable PPS reference

Long Haul over WR BiDi DWDM

▪ Fixed latency of WR equipment calculated using WR Model

▪ Resolves calibration problems of tech and “complex” OF networks:

▪ Uncertainty of the wavelength long distance BiDi SFPs

▪ Uncertainty of chararterisitics among OF vendor

▪ α in networks with mixed version of OF (G652b/d)

▪ Reduces impact of αDWDM ~10-6 < αBiDi ~10-4

Long Haul over WR BiDi DWDM

▪ Resolves calibration in mixed FO networks G652b and G652d

Device SFP Fiber Optic α Offset (ps)

WRS BiDi 50 km G652b 2.59E-04 225

WRS BiDi 50 km G652d 2.59E-04 2860

WRS BiDi 20 km G652b 2.36E-04 1158

WRS DWDM 50 km G652b 3.80E-06 11

WRS DWDM 50 km G652d 3.80E-06 8

WRS DWDM 20 km G652b 3.80E-06 52

WRS DWDM 100 km G652b&d 3.80E-06 46


www.sevensols.com

White Rabbit topologies

White Rabbit network topology

WR good practices.pdf


• Time server: It is the device that the White Rabbit network uses as the time reference. This time reference will be distributed through the White Rabbit network with sub-nanosecond accuracy. Using a time server is optional, if no time server is connected to the White Rabbit network the top White Rabbit device will work as a free-running mode. The time server must provide 1PPS and 10 MHz signals to the first White Rabbit node, which will lock to the incoming frequency and will determine the second beginning with the 1PPS input. Additionally, a NTP server can be configure to give a notion of time to the first White Rabbit node. Standard PTP could be used in specific conditions to substitute the 1PPS/10 MHz and NTP from the time server.

• First White Rabbit device: This device must be configured as Grandmaster. In this mode it will lock to the incoming 10 MHz signal and will determine the second beginning with the 1PPS input. All its optical interfaces will be automatically configured as master, so this device can distribute the obtained time reference to the subsequent White Rabbit devices.


• Intermediate White Rabbit devices: These devices receive the time reference using the White Rabbit protocol on one of their optical interfaces. They can daisy-chain this White Rabbit reference on other optical port, acting as master of the lower level. In the case of WR-Switches, they can deploy a tree topology using one of their optical ports to receive the White Rabbit reference from the first White Rabbit device and the other 17 optical ports to distribute it to other White Rabbit compliant devices.

• Last White Rabbit device: This device receives the time reference from a previous White Rabbit device and it is responsible of propagating this time reference to any other device which will be synchronized. There are several interoperability options depending on the White Rabbit device that is used on this last hop:

• IEEE 1588, Standard PTP v2. • 1PPS synchronization. • White Rabbit integration in third party devices. • NTP. • IRIG-B, NMEA, ToD…



White Rabbit network topology: Local scenarios




White Rabbit network topology: Remote scenarios





When every nanosecond counts White Rabbit training · White Rabbit technology: Time reference Atomic clocks count time relying on the resonance frequency of atoms excited by microwave,

Documents