DUAL-OUTPUT HOLA FIRMWARE AND TESTS Anton Kapliy Mel Shochet Fukun Tang Daping Weng.

DUAL-OUTPUT HOLAFIRMWARE AND TESTSAnton Kapliy

Mel Shochet

Fukun Tang

Daping Weng

2

Summary• New dual-output HOLA board design with independent flow

control from DAQ and FTK channels• See Fukun Tang’s slides for a detailed walkthrough

• New firmware to interface the S-LINK core to inexpensive Altera Cyclone IV GX transceivers• 90% of original HOLA code preserved.• Fitting & assembling performed in Quartus 10.1 SP1

• We also have an experimental port to Quartus 11.0 (released in May 2011)

• Simulation using a comprehensive Modelsim testbench

• Manufactured, stuffed, and tested two prototype boards:• Firmware loaded on power-up from a 4 Mbit EPROM (EPCS4)• Data sent through both channels• Ensured that flow control can be asserted from either channel• BER tests using 1.35E15 bits sent through both channels

3

What is a HOLA?

A device that implements the S-LINK protocol for point-to-point data transfer.

Pixel and SCT data in a ROD is organized into 32-bit chunks and sent to a HOLA card, which serializes and sends it through optical fibers.

Another HOLA card receives this data stream in a ROB (and FTK DF) and de-serializes it.

S-LINK implements data framing, error detection, and flow-control.

4

FIFOD[32] @ 40 MHz D[32] @ 50 MHz

Interface to TLK

TLK2501 chip(sends data)

FPGA

TLK2501 chip(receives data)

FPGA provides a parallel interface to an outside SERDES device (TLK-2501), which feeds serial signal to an optical transmitter.

However, TLK-2501:• Is deprecated and not marketed by TI anymore• Costs more than a Cyclone IV FPGA• Consumes four times as much power

D[16] @ 100 MHz

Big picture: original HOLA LSC

(only forward channel shown)Mezzanine data port

TX_ENTX_ER

Serial connection(via optical transceiver)

Big picture: new HOLA LSC

5

FIFOD[32] @ 40 MHz D[32] @ 50 MHz

Interface to TLK

ALTGX wrapper(emulates TLK2501)

FPGA

Serial connections(via optical transceivers)

(only forward channel shown)

Xilinx FPGA(FTK DF)

TLK2501 chip(DAQ)

OLD LSC CORE

(Altera SERDES megafunction)

TX_CLK

90% of original LSC code is unchanged.ALTGX wrapper emulates TLK2501:• “Plugs in” into the original LSC core code• Implements link startup and synchronization

• TLK receiver on DAQ side is oblivious to the change

D[16] @ 100 MHz

Mezzanine data port

TX_ENTX_ER

6

Modelsim testbench

Stefan Haas gave us a Modelsim testbench implementing the above configuration.We updated some features of this testbench to fit our purposes:• Serial LSC <-> LDC communication is simulated using Altera transceivers• LSC gains another channel (thus, “dual-output” HOLA)• Another LDC+ROMB block was added to receive data from the second channel.FEMB, LDC, and ROMB were simulated functionally, while LSC (including Altera transceiver interface) - using a compiled & fitted (gate-level) Altera Quartus design.

ROD DAQ/FTK ROB

7

PC setup: ROD sideS32PCI64 “SOLAR” mezzanine card:Provides access to S-LINK via PCI bus

The first prototype of dual-output HOLA

Mezzanine connector to transfer 32-bit data

2 x optical transceivers

FEMB S-LINK LSC

8

PC setup: ROB and FTK side“FILAR” card (based on S32PCI64):• Implements S-LINK protocol• Provides access to S-LINK via PCI bus

S-LINK LDC and ROMB (in one package)

FTK

DAQ

Identical data is received from the two fibers running from the dual-output HOLA. Note that each channel can assert flow control.

The two channels are read out independently. E.g., we can read out all pending buffers from DAQ, so that only FTK will be asserting flow control.

Multi-mode fibers Optical attenuators(7 db @ 850 nm)*

* We also ran BER tests to ~2E14 bits using attenuators rated for 10 db @ 1310 nm

9

BER test with 1.35E15 bits• We use the so-called SLIDAS mode in SOLAR PCI card• For the main BER test, we used the following configuration:

• Continuous frames with 254 words x 32 bits are fed into HOLA LSC• These words follow a pseudo-random sequence reproducible in C++

• On reception, we check word-by-word that all received patterns are correct

• PC presents a bottleneck: need to generate these pseudo-random patterns and compare them with received data words.• We can achieve about 65 MB/s (21 days to test BER to 1E15).

• Under this scheme, we continuously exercise flow control:• FEMB wants to send data at nearly 200 MB/s, but ROMB & LDC buffers

quickly fill up and assert flow control until they are read out and “recharged” by the slow PC

• Effectively, flow control is asserted / de-asserted continuously

• 1.35E15 bits were received by both FILAR channels without any error (~4 weeks of non-stop transmission)

10

Additional tests• After the BER test was completed, Daping spent a couple

of weeks running additional tests:• Generating data on the PC (rather than using SLIDAS patterns)

• Send data to the SOLAR card via the first PCI port• Read it back on reception from the FILAR via the second PCI port.

• SLIDAS random data with 16-1024 words per frame• Theoretically, the frames can be even larger, but our FILAR device

drivers are hardcoded to allocate enough memory only for 1024 words.

• ROD mode• Specially-designed frames that look like LHC data• Each frame contains a header, payload, and a footer

• TEST mode (a walking-bit pattern)

• All tests have finished without any errors

11

BACKUP SLIDES

12

SLIDAS ROD mode• Data pattern:

• There is one control word at the beginning of each frame

• Next is a header of length 9 words, containing information indicating that this is the Nth frame sending

• The header is followed by an increasing arithmetic progression starting from 0 and ending at (frame length – 15)

• Next is a footer of length 3 words• Ending control word

• Length options are the same as in the random data mode

• We tested to the order of 1E9 words, 256 words per frame and no error was observed

A Typical Frame:

Control word399417060632336855040N (mark of the Nth frame)3N-2 mod 40960000123...Frame length – 151Frame length – 140Control word