FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group

Upgrades of detector readout meeting, 9 July 2014 1

FELIX Design

Upgrades of detector readout meeting9 June 2014

Lorne Levinson, for the FELIX group

L. Levinson

Upgrades of detector readout meeting, 9 July 2014 2

Guiding principles• Goal: a GBT to LAN “switch” with as little awareness of detector specifics as

possible– No user code in FELIX (so far): configuration params for options

• Separate data movement from data processing• Configurable data paths from E-link to network endpoint• Scalable at Front end, Back end, and switch levels• Do as much as possible in software• COTS FPGA PCIe board

– But… see talk on TTC clock recovery by Jinlong Zhang • Readout, so latency is not important,

but provide low latency GBT-to-GBT links• Future proof by following COTS market, separately upgradeable components:

GBT, FPGA PCIe board, serve PC, network cards• Routing is for E-links; they are the logical connections

L. Levinson

Upgrades of detector readout meeting, 9 July 2014

3

Virtex-7 PCIe board

L. Levinson

From Global HiTech, PCIe gen 3, 8 lanesVirtex X690T2x12 bidir CXP connectors, FMC

Upgrades of detector readout meeting, 9 July 2014 4

Off- detector architecture with FELIX

• Functions can now be separated, physically, or just logically.

• “ROD” = Event fragment builder/processor for ROS

L. Levinson

“ROD”

Upgrades of detector readout meeting, 9 July 2014 5L. Levinson

Upgrades of detector readout meeting, 9 July 2014 6

Routing• Configured via control port on main or separate network

• Simple/fixed:FELIX port # / E-link-range → IPaddress / portor → Infiniband local identifier / queue pair

• E-link packet protocol: – Allows several logical streams to use the same E-link header with: stream-id + packet len (if not fixed length packets)FELIX port # / stream-id → IPaddress / port…

• Cloning, broadcasts / multicasts possible, in either direction

• Reliable transport from FELIX to end-point via TCPIP

• Configurable Quality of Service per logical connection– Less-than-best-effort for event data for monitoring

L. Levinson

Upgrades of detector readout meeting, 9 July 2014 7

Use cases – 11. Receive data from an E-link or bonded group of E-links (a channel) and

route it to one or more network endpoints via TCP or UDP. Bits must be arranged into correct order. – Optional 8b/10b or HDLC checking and/or decoding – Optionally verify checksum – Packet boundary identification either by SOP/EOP or length, optionally

there can be no packet boundary, just a byte stream. • If a maximum packet length is configured, data arriving after the

maximum is discarded and this error is marked in the packet status word.

– If streams are defined for this channel packets are routed according to the stream. The stream ID may contain an extension filled by the detector with part of the Level-1 ID or the Trigger Type for advanced event routing.

– Sampling can be requested

L. Levinson

Upgrades of detector readout meeting, 9 July 2014

8L. Levinson

Upgrades of detector readout meeting, 9 July 2014 9

Use cases – 22. Receive BUSY from an E-link command, internally from FELIX itself, or

from the network. Then route it to a network endpoint or to a standard ATLAS RODBUSY signal (open collector Lemo). In Phase II it may be sent to the TTC system.

3. Receive TTC information (BC, L1A, ECR, BCR, user bits) and broadcast them to the requested E-links – The BC clock is supplied to the GBT because the FELIX's GBT link clock

is the BC clock supplied by the TTC system to FELIX. – Three options: 2-bit, 4-bit, 8-bit as described in the datasheet.

4. Receive data from a TCP port and route it to one or more E-links – Optional 8b/10b or HDLC encoding – A specific multicast channel can broadcast packets to all out-going E-

links with the "MC" attribute.

L. Levinson

Upgrades of detector readout meeting, 9 July 2014 10

Use case – 3

8. Scheduled data transfer lists: setting and activating (list is per GBT, but E-links are enabled by a bit map.) Open Issue: Or, several lists per FELIX and each E-link configures to use a specific list. Each item in the list is qualified by a delay to wait from the previous item in the list. – Activation by TTC user bit – Activation by command received via TCP

9. Receive data from an E-link or bonded group of E-links (a channel) and route it to a Direct Output Low Latency link (probably a GBT). Typically only a particular stream in the channel would be so routed.

L. Levinson

Upgrades of detector readout meeting, 9 July 2014

11L. Levinson

Low latency path is between predefinedGBT pairs.Uses “cut-through”Routing.

Upgrades of detector readout meeting, 9 July 2014 12

FELIX: FW/SW interface

Data interface• PCIe low level packet format (fixed length)• PCIe high level packets, spans low level packets• Push/pull? Hints from RobinNP architecture

Configuration parameter interface• Need description of parameters that can generate:

XML for OKS, VHDL for FPGA• See (but needs to be updated):

https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf

Monitoring and statistics interface

FPGA configuration interface• IP core from HitechGlobal ~4k$

L. Levinson

Upgrades of detector readout meeting, 9 July 2014

13L. Levinson

Upgrades of detector readout meeting, 9 July 2014

14L. Levinson

Upgrades of detector readout meeting, 9 July 2014

15

Backup

L. Levinson

Upgrades of detector readout meeting, 9 July 2014 16

Functionality of E-lane handlersFrom GBT: need to decode 8b/10 or HDLC data• Collect 2, 4, 8, 16 bits per 40MHz clock• Align using comma symbols• Decode 8b/10b, discard comma, Start-of-Packet, End-of-Packet• Add 1KB block headers and “chunk” trailers• Optionally verify checksum

To GBT: need to encode 8b/10b or HDLC data• Read from FIFO• Encode (8b/10b(?) or HDLC) 2 bytes at a time from FIFO• Chop into 2, 4, 8, 16 bits per 40MHz clock

Probably run at 80MHz to get extra clocks for headers/trailers

L. Levinson

Upgrades of detector readout meeting, 9 July 2014 17

Clock recovery

• The 2x40MHz clock comes from the TTC link, i.e. a recovered clock.• Probably OK for GTH’s in the same GTH block of 4 transceivers• ???

– Can it be distributed to the other blocks?– Does the recovered clock have sufficiently low jitter for the other

GTH’s?

• We should probably consider a TTC receiver and jitter remover on an FMC card. – rework the GLIB version so that the FMC is on top of the PCIe mother

board instead of extending beyond the PCIe board boundary

L. Levinson

Upgrades of detector readout meeting, 9 July 2014 18

Error handling

• Just beginning the considerable thought that needs to be done here.• Two issues:

– Discovery of corrupted data– Surviving errors without out loosing event boundaries

• This requires some effort on the front ends• Corruption within a GBT 80-bit word is detected by the GBT’s FEC• Hard to believe E-links can be mixed into each other,

so expect errors within E-link channels: repeated clocks, lost clocks

• If E-links use 8b/10b with packet framing codes, fixed length, or variable length with counts, then errors (not all) can be detected.

• FELIX can flag these errors in the header it sends with a stream’s data

L. Levinson