PDF (6.16 MB) - IOPscience

Journal of Instrumentation

A PCI Express optical link based on low-costtransceivers qualified for radiation hardnessTo cite this article A Triossi et al 2013 JINST 8 C02011

View the article online for updates and enhancements

You may also likeAssessing the safety of express bus frompassengersrsquo perspectiveA H Azman S Abdullah S S K Singh et al

-

Research on optimization of trafficmanagement and control measures forexpressway emergency based on VISSIMsimulationLi Jiahui Wu Keman Li Yudian et al

-

(Invited) the Progress and Outlook of GaNLaser DevicesJung Han

-

Recent citationsDigital Front-End Electronics for theNeutron Detector NEDAF J Egea Canet et al

-

This content was downloaded from IP address 21910037239 on 29112021 at 1051

2013 JINST 8 C02011

PUBLISHED BY IOP PUBLISHING FOR SISSA MEDIALAB

RECEIVED November 15 2012ACCEPTED December 26 2012

PUBLISHED February 5 2013

TOPICAL WORKSHOP ON ELECTRONICS FOR PARTICLE PHYSICS 201217ndash21 SEPTEMBER 2012OXFORD UK

A PCI Express optical link based on low-costtransceivers qualified for radiation hardness

A Triossia1 D Barrientosbcd MBellatob D Bortolatoa R Isocrateb G Rampazzob

and S Venturab

aIstituto Nazionale di Fisica Nucleare ndash Legnaro National LaboratoryViale dellrsquoUniversita 2 35020 Legnaro (PD) Italy

bIstituto Nazionale di Fisica Nucleare ndash PadovaVia Marzolo 8 35131 Padova Italy

cIFIC CSIC-Universitat de Valenciac Catedratico Jose Beltran 2 46980 Paterna Spain

dDepartamento de Ingenierıa Electronica Universitat de ValenciaAvinguda de la Universitat sn 46100 Burjassot Valencia Spain

E-mail atriossipdinfnit

ABSTRACT In this paper we want to demonstrate that an optical physical medium is compatiblewith the second generation of PCI Express The benefit introduced by the optical decoupling ofa PCI Express endpoint is twofold it allows for a geographical detachment of the device and itremains compliant with the usual PCI accesses to the legacy IO and memory spaces We proposetwo boards that can bridge the PCI Express protocol over optical fiber The first is a simple opticaltranslator while the second is a more robust switch developed for connecting up to four devices toa single host Such adapters are already working in the control and data acquisition system of aparticle detector at CERN and hence they had been qualified for radiation hardness The positiveoutcomes of the radiation tests of four types of off-the-shelf transceivers are finally reported

KEYWORDS Data acquisition circuits Radiation-hard electronics Detector control systems (de-tector and experiment monitoring and slow-control systems architecture hardware algorithmsdatabases)

1Corresponding author

ccopy 2013 IOP Publishing Ltd and Sissa Medialab srl doi1010881748-0221802C02011

2013 JINST 8 C02011

Contents

1 Introduction 1

2 LINCO boards 221 Optical adapter 322 One-to-four optical switch 4

3 Radiation test 531 Test setup 532 Test results 6

321 Infineon V23848-N305-C56 6322 Intel TXN31115D2 7323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 7

33 PCI Express compatibility 8

4 Conclusion 8

1 Introduction

Over the past decade we have seen a definite stand out of PCI Express [1] as the de-facto standardprotocol for host to peripheral interfaces It allows devices to achieve very high data transfer rateimproving the bandwidth steadily every time a new generation appears on the market Clearlyusing copper cabling the maximum reachable physical distance of a remote IO device decreaseswith the increase of the data rate Thus industry leaders in PCI Express solutions and in fiber opticproducts are moving towards a new off the record standard for the PCI Express physical layer [2]We want to contribute to such research topic carrying on with our preliminary results accomplishedin the context of the LINCO project [3] We developed a bus adapter able to bridge remote buses(gt 100 m) to a single-host computer giving the legacy PCI compatibility to the endpoint device andwithout even the need of a specialized driver Furthermore the adapter was made tolerant to harshenvironmental conditions like strong magnetic fields or radiation fluxes requirements that the dataacquisition of high-energy physics experiments often face

The adoption of the LINCO adapter in several physics experiments like AGATA [4] WArP [5]or CMS at LHC [6] demanded an improvement in the hardware design in order to exploit thefull bandwidth of the high speed PCI Express protocol and in order to achieve full decouplingbetween remote devices and the local hosts keeping the transparency of the link With the aimof facing these two specific requirements we developed two new models of LINCO board thatwill be described in section 2 They were designed around the PCI Express features that highlightthe communication protocol nature of the PCI Express IO bus Indeed the protocol provide forload-store operations between two nodes performed by exchanging framed packets in accordance

ndash 1 ndash

2013 JINST 8 C02011

to a suite of stacked protocol layers taking care of the physical link and transaction issues of thechannel By only replacing the physical layer with an optical one we want to keep the model offield bus control in which a host and a networked peer node exchange software arranged packets toaccess memory and registers of the field bus for IO operation

All the LINCO boards are used in the Detector Control System (DCS) and local Data Ac-quisition System (DAQ) of the muon drift tube chambers employed in the trigger of the CMSexperiment Hence a major consideration in the design of the link is the performance in presenceof radiation The remote part of the link resides in electronic crates that will be exposed to an in-tegrated proton fluence of 5 middot1011 pcm2 (corresponding to a total ionizing dose of about 66 Krad)on the components surface equivalent to over 10 years of LHC running Early performance testsof the candidate link and irradiation studies [3] have been accomplished dividing the board in twodifferent areas one involving the PCI-PCI Express bridge and the clock generation and filteringsystem and the other only the optical transceivers No single event upset (SEU) was observed buta series of micro latch up events in the transceivers electronics became destructive at a total radia-tion dose of 1 krad although the link was based around VCSELs (Vertical Cavity Surface EmittingLasers) that were chosen for their radiation tolerance [7] During the whole life span of CMS ex-periment devices are generally biased during the irradiation periods therefore it is important toexamine the transmission life parameter keeping the circuits biased and working during the fulltest A detailed study of the transceivers performance in presence of proton radiation is thereforesubject of this paper In section 3 four kind of commercial low-cost Small Form Factor Pluggable(SFP) transceivers will be compared Finally an accurate jitter analysis to check the PCI Expressrequirements will validate the effectiveness of the optical link at the Bit Error Ratio (BER) levelsettled by specifications

More precisely in this paper we provide the following novel contributions

bull Manufacturing of a PCI Express optical translator able to exploit the full bandwidth theprotocol can offer

bull Manufacturing of a general purpose one-to-four PCI Express Gen2 optical switch that canbe used as simple fan-out generic data mover or for distributed computing and

bull Improved testing on the proton radiation tolerance of the laser part in order to ensure PCIExpress compatibility up to 5 middot1011 pcm2

2 LINCO boards

The LINCO board comes in three flavors a PMC mezzanine a native PCI Express version and aone-to-four PCI Express switch The first adapter (described in [3]) was centered around a PCI-PCIExpress bridge and it exploited a couple of optical transceivers to transmit the data and the clocksignals To use up the full bandwidth of the link and to be up-to-date with native PCI Expressmotherboards a bridge-less version of the board was manufactured In the next section the boarddescription and results from a bandwidth test are reported In section 22 the last version of LINCOis presented it embeds a PCI Express switch that allows the collection of up to four data lanes andgrants more flexibility to the reference clock configuration

ndash 2 ndash

2013 JINST 8 C02011

(a)

Pc

iEx

1x

Fin

ger

CPLD

Filter

SFP Transceiver

SFP Transceiver

Buffer

PCI Ex x125GTs

Perst

REF CLK

(b)

Figure 1 (a) PCI Express optical adapter (b) block diagram

21 Optical adapter

The PCI Express optical adapter shown in figure 1(a) is a x1 lane PCI Express optical trans-lator with an edge connector that allows the insertion in a standard PC motherboard Two SFPtransceivers are assigned to optically convert the data and clock differential pairs (figure 1(b)) APhase-Locked Loop (PLL) with a small loop bandwidth (ICS8741004AIL) is used to filter the jit-ter of the reference clock and to provide a signal suitable for feeding the related transceiver Thedifferential signal received by such transceiver is routed to an Altera programmable logic device(CPLD) with the purpose of detecting the presence of a remote device if it loopbacks the clocksignal The CPLD is also in charge of taking care of the PCI Express receiver detect mechanismpower cycling the Pericom PI2EQX4401 buffer when it receives a PCI Express reset signal andswitching off only the receiving channel when the remote device is not sending valid data

Special attention was paid to the power supply system as potential power integrity problemsrelated to the high-speed signals (signal rise time is approximately 70 ps) may appear An increaseof jitter whose budget is strictly limited could rise from an excessive power related noise SiWavesoftware by Ansoft Corporation was first used to spot potential power or ground bounce and toanalyze DC voltage distributions and finally to simulate the electromagnetic field on the high-speed traces and to obtain the characteristic S-parameters to be used in a Hspice program

To test the bandwidth limit of the PCI Express protocol transmitted on optical medium wedesigned a DMA engine that feeds a PCI Express core instantiated on an Field ProgrammableGate Array (FPGA) This commercial IP directly drives the Multi-Gigabit Transceivers (MGTs)required for PCI Express communication and implements all the protocol staked layers up to thetransactional one To carry out the test we chose a Xilinx Virtex4 evaluation board with SFPoptical transceivers in order to establish the optical link with our adapter which was plugged in aPCI express slot of a personal computer We used a simple driver charged to initially allocate a200 MByte of non-contiguous main memory and afterwards to request the DMA transfer and tomanage the interrupts notification In this way an user program can ask the driver for a big chunkof the PCI device memory recovering back after the DMA transfer completed a virtual memoryvector pointing on the required data The maximum payload available by the core (128 Byte) wasquite far from the 4 KBytes foreseen by the PCI Express specifications however we could achievea top transfer rate of 200 MBytes per second

ndash 3 ndash

2013 JINST 8 C02011

(a)

MuxDemux PI2PCIE2412

PLX Switch PEX 8609

P

ciEx

4x

Fin

ger

MuxDemux PI2PCIE2412

SFP Transceiver

SFP Transceiver

SFP Transceiver

SFP Transceiver

CLK Fan-out PI6C20400S

CLK Fan-out PI6C20400S

100 MHz Oscillator

FPGA Spartan3AN

EEPROM

Power Manager

POWR1014A

MOSFET

DC-DC DC-DC

DC-DC

PCI Ex x45GTs

REF

CLK

PERST

I2C

Differential

SEL1

SEL0

MOSFET

DC-DC

(b)

Figure 2 (a) One-to-four optical switch (b) block diagram

22 One-to-four optical switch

The card is based on a PLX switch (PEX8609) that has the upstream port (x4 lanes) routed to thePCI Express finger connector (figure 2(a)) The downstream ports can be configured by a smallon board FPGA (Xilinx Spartan 3AN) in two possible modes of operation four one-lane ports orone four-lane port The FPGA is also charged to configure the clock domain and hence the clockpath in one of the following options

bull One clock domain the reference clock of the local PCI Express bus is broadcast to the switchand to two transceivers

bull Two clock domains the reference clock is used only by the upstream port of the switch anda plane 100 MHz clock is broadcast to the downstream ports

The same physical routes towards the optical transceivers can be used by clocks or data lanesdepending on the selected path by the configuration of two multiplexerdemultiplexer (PericomPI2PCIE2412) In figure 2(b) the block diagram of the board is shown

It is worth noting that we can avoid to send the reference clock to the remote devices if theSpread Spectrum Clock (SSC) is not active on the upstream port since the clock can be recoveredfrom the datastream In such case we should not be worried about Electromagnetic Interference(EMI) because there is a reduced need of emissions suppression since the links are optical

Due to the embedded features of the PLX switch the board can be used in several applicationsThe four Gen2 PCI-Express lanes offer an aggregated maximum bandwidth of 20 GTs suitable forhigh throughput DAQ systems Furthermore the integrated DMA engine removes the burden re-sulting from moving data between devices away from the processor The board can fit in distributedcomputing DAQ systems as well a downstream port can be configured as non-transparent in orderto isolate host memory domains by presenting any processor subsytems as a simple endpoint ratherthan a complete memory system

We have performed extensive jitter tests on the receiver side as specified by the second revisionof the PCI Express base specification [9] in order to verify that even with the introduced jitter in

ndash 4 ndash

2013 JINST 8 C02011

(a) (b)

Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps

the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered

3 Radiation test

In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI

We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed

31 Test setup

The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose

ndash 5 ndash

2013 JINST 8 C02011

(a) (b)

Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising

of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device

32 Test results

In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below

321 Infineon V23848-N305-C56

The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region

ndash 6 ndash

2013 JINST 8 C02011

(a) (b)

Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose

shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend

The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps

322 Intel TXN31115D2

We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency

323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2

These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive

ndash 7 ndash

2013 JINST 8 C02011

nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010

and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged

to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission

33 PCI Express compatibility

Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram

(BER(t))i = ρT

(int +infin

t(PDFT (τ))i dτ

prime+int t

minusinfin

(PDFT (τ))i dτprime)

(31)

Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits

In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)

4 Conclusion

The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints

ndash 8 ndash

2013 JINST 8 C02011

for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections

A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel

In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market

References

[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom

[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper

[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518

[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]

[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]

[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004

[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795

[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)

[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom

[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper

ndash 9 ndash