Peter Jansweijer MROD Design Review: November 12, 2003 Slide 1 MROD-1 Hardware Overview • MRODin • MRODout • MROD-1 = 3 x MRODin + 1 x MRODout
Jan 24, 2016
Peter JansweijerMROD Design Review: November 12, 2003Slide 1
MROD-1 Hardware Overview
• MRODin
• MRODout
• MROD-1 = 3 x MRODin + 1 x MRODout
Peter JansweijerMROD Design Review: November 12, 2003Slide 2
Memory
SHARC
MemoryFPGA
VME64x
TTCinterface
MRODout
MRODin
FPGA
Memory
SHARC
MemoryFPGA
FPGA
Memory
SHARC
MemoryFPGA
FPGAFPGA
1
2
4
3
1
2
4
3
1
2
4
3
SHARCA
1
2
43
0
SHARCB
1
2
4
3 0
5
5
LVDSIn
LVDSOut
SHARCLinks
C
D
E
MRODin
MRODin
6xS-Link or
GOLA
MROD-1 Hardware Overview
Peter JansweijerMROD Design Review: November 12, 2003Slide 3
MRODin• Buffer memory, Separate partition per TDC,
Dataflow into the buffer memory• Tetris register, I2O-FIFO, Fault tolerance, Error
signaling and handling , Resynchronization• Dataflow out of buffer memory, DMA, Zero
suppression, Length FIFO• Data format, TDC-ID, Parity errors• Interrupt sources• FPGA registers
Peter JansweijerMROD Design Review: November 12, 2003Slide 4
1 MBZBT Buffer
Memory
SHARC
Data FIFO
TetrisRegister
Input
Output
I2OFIFO
Control
Control/StatusError signaling
4 Sharc links40 (80) MB/S
Each
FIFO
Length FIFO
FPGA Channel A
FPGA Channel B
MRODin OverviewS-Link
orGOLA
S-Linkor
GOLA Channel BMemory
Peter JansweijerMROD Design Review: November 12, 2003Slide 5
MDT Readout chain wooden model
Peter JansweijerMROD Design Review: November 12, 2003Slide 6
Buffer Memory PartitionsDataflow into buffer memory
S T N T T S T T T T
Time Division Multiplexed
S T T T T N S T N
TT T
T
TT
T TTT
TT
1 20 3 4 5 1 20 3 4 5 1 20 3 4 5 1 20 3 4 5
TD
C 0
TD
C 1
TD
C 2
TD
C 3
TD
C 4
TD
C 5
S
N
Separator = Start new TDM Cycle
No Data = Time slot filler
In real life: 18 Buffer Memory Partitions (8 K words each)
TTDC data word TDC trailer
Note:TDC headers are omitted for simplicity!
Peter JansweijerMROD Design Review: November 12, 2003Slide 7
Tetris RegisterNormal operation
TD
C 0
TD
C 1
TD
C 2
TD
C 3
TD
C 4
TD
C 5
Example Tetris Register = 6 x 4In real life, Tetris Register = 18 x 16
ExpectedEvent-ID
n
n+3
n+1
n+4
I2O-FIFO
All trailers ofEvent-ID n
arrived
Event-ID n
Peter JansweijerMROD Design Review: November 12, 2003Slide 8
Tetris RegisterTDC trailer missed
TD
C 0
TD
C 1
TD
C 2
TD
C 3
TD
C 4
TD
C 5
ExpectedEvent-ID
n
n+3
n+2
n+5
I2O-FIFO
All trailers ofEvent-ID n+1
arrived
Event-ID nEvent-ID n+1
Peter JansweijerMROD Design Review: November 12, 2003Slide 9
Tetris RegisterNotorious absence of TDC
(panic mode)
TD
C 0
TD
C 1
TD
C 2
TD
C 3
TD
C 4
TD
C 5
ExpectedEvent-ID
n
n+3
n+1
n+4
I2O-FIFO
Arrival of a trailerfor Event-ID n+3
Event-ID n
Peter JansweijerMROD Design Review: November 12, 2003Slide 10
Tetris RegisterEarly / Late
TD
C 0
TD
C 1
TD
C 2
TD
C 3
TD
C 4
TD
C 5
ExpectedEvent-ID
n+1
n+4
n+1
n+4
Early TDC1 trailer
Late TDC 1 trailer
Notes:• Expected Event-ID can be set by the SHARC during MRODin initialization• Expected Event-ID is hardware updated• Resynchronization for an MRODin channel should be possible.
Peter JansweijerMROD Design Review: November 12, 2003Slide 11
Buffer Memory PartitionsDataflow out of buffer memory
TT T
T
TT
T TTT
TT
TD
C 0
TD
C 1
TD
C 2
TD
C 3
TD
C 4
TD
C 5
Read fromI2O-FIFO:
TDC Link Present(TLP) word 10 2 3 4
T T T T T
5
T
Trailer and Word Count (TWC)Word Count and Event-ID
To Output Data FIFO (128)
Event-ID n
To Length FIFO (128)
• When a TDC is not present then don’t read the partition! You’ll never encounter a Trailer!
• This information is signaled in the data stream (TLP word)
Peter JansweijerMROD Design Review: November 12, 2003Slide 12
• Output Data FIFO is read with continuous DMA by the SHARC
• Length FIFO provides event summary
• While reading TDC data from buffer memory, data can be zero suppressed(i.e. TDC header immediately followed by TDC trailer)
Dataflow out of buffer memoryContinued
Peter JansweijerMROD Design Review: November 12, 2003Slide 13
When the MRODin encounters a TDC Header, it reformats the TDC-ID bits:
Data Format:TDC-ID in TDC Header word
31-28 27-24 23-0
TDC-ID (4-bit)Upper Nibble = TDC data type0x1010 TDC Header0x1011 End Of TDC Group (used by AMT-1;
Not used by AMT-2/3 chip)
TDC wordreceived by CSM
31-29
5-bit TDC-ID based on TDM time slotTDC Header
28-24 23-0
With 5-bits, 18 TDCs can be coded
See: T. Wijnen, “The MROD data format”, (ATL-COM-MUON-2003-011)http://www.hef.kun.nl/atlas/
Peter JansweijerMROD Design Review: November 12, 2003Slide 14
31-28TDC wordreceived by MROD
27 26 25-24 23-0
2 bits, residue from TDC-IDCSM -> MROD Parity bitTDC -> CSM Parity ErrorUpper Nibble = TDC data type
31-28 27-24 23-0
TDC-ID (4-bit)Upper Nibble = TDC data type
TDC wordreceived by CSM
31-28
Error Replacement Code
27-24 23-0
Data Format: Parity errors
Upper Nibble
When the MRODin encounters a TDC Parity Error or an Input Link Parity Error,it replaces the upper nibble with an Error replacement code (programmable)
Peter JansweijerMROD Design Review: November 12, 2003Slide 15
MRODin Interrupts• IRQ0 X I2O_FIFO Full (128 entries)
X S Buffer Memory Partition Full (8 K words)
S Read Out MaximumTDC Parity error (+ Overrun)
• IRQ1 Input Link Parity error (+ Overrun)Input Link Down
• IRQ2 TDC Early, Late (+ Overrun)
Notes: X: Fatal
S: Shutdown Read-out for particular TDCAll interrupt sources are individually mask-able
Peter JansweijerMROD Design Review: November 12, 2003Slide 16
MRODin FPGA Registers• Separator
– Pattern– Control Bit Pattern– Mask– Control Bit Mask
• TDC Header– Pattern– Control Bit Pattern– Mask– Control Bit Mask
• TDC Trailer– Pattern– Control Bit Pattern– Mask– Control Bit Mask
• No Data– Pattern– Control Bit Pattern– Mask– Control Bit Mask
• Error-Code Replace Patterns
• msb’s MRODin Header Pattern (TLP)
• msb’s MRODin Trailer Pattern (TWC)
• Event Length FIFO• Interrupt Control IRQ0
– I2O_FIFO Full
– Buffer Memory Partition Full
– Read-out Maximum
– TDC Parity Error (Overrun)
• TDC Parity Error Individual Interrupt Mask
• Input Link Interrupt IRQ1
– Input Link Parity error (Overrun)
– Input Link Down
• Early and Late Event-ID IRQ2
– TDC number, Event-ID (Overrun)
• Test & Input Link Control Status Register
• Test Link Data Register• Test Link Control
Register• Maximum Event Size• Expected Event-ID• TDC Mask Register• Partition Read-out Enable• Separator Flags Register
Peter JansweijerMROD Design Review: November 12, 2003Slide 17
• Event building of TDC data
• Tetris register creates a fault tolerant design
• FPGAs supply SHARCs with appropriate (error) information
• Testable by SHARC via access to registers and memory
Summary MRODin
Peter JansweijerMROD Design Review: November 12, 2003Slide 18
• VME64x Interface• TTC Interface via TIM• FIFO and Flow control via ROL• Interrupt sources• FPGA registers
MRODout VME64x
TTCinterface
MRODout
FPGA
SHARCA
1
2
43
0
SHARCB
1
2
4
3 0
5
5
LVDSIn
LVDSOut
SHARCLinks
Peter JansweijerMROD Design Review: November 12, 2003Slide 19
VME64x Slave Interface VME64x
Guideline; Chris Parkman, “ATLAS Read Out Driver VMEbus Implementation”:http://atlas.web.cern.ch/Atlas/GROUPS/FRONTEND/documents/ROD_VME83.pdf
• CR/CSR (AM 0x2F)– A24/D32 Single Cycle Mandatory– A24/D32/D16/D08(EO)/D08(O) Single Cycle, RMW Optional
• SHARCs– A32/D32 Single Cycle, RMW, BLT Preferred
• CR full VME64x range, currently filled with the VME64 subset• CSR BAR/BitSet/BitClr registers• SHARC may generate a VMEbus Interrupt
– Selectable IRQ level [1..7] {I(n) D08(O) where n=1..7} – Programmable 8-bit Status-ID– ROACK Preferred
• AM 0x10 (User Defined) used as HARD Reset for the SHARCs
Peter JansweijerMROD Design Review: November 12, 2003Slide 20
TTCinterface
TTC Interface
TIM Designed by University College Londonhttp://www.hep.ucl.ac.uk/atlas/sct/tim/
• Uses a special P3 backplane with a so called TTC-bus (8 serial signals)
• TIM Distributes:– Event-ID / Bunch-ID Put into a FIFO that can be read by the
SHARC
– Trigger-Type Put into a FIFO that can be read by the SHARC
– ECR Connected to SHARC IRQ1
– BCR not used by MROD
– L1A not used by MROD
• TIM Incorporates Busy Logic to throttle the CTP
Peter JansweijerMROD Design Review: November 12, 2003Slide 21
FIFO and Flow control via ROL
• Output S-Link interface 160 MB/s
• FPGA contains S-Link Output FIFO (256) to avoid local bus stall due to an XOFF
• SHARCs can transfer output data through a chained DMA
Peter JansweijerMROD Design Review: November 12, 2003Slide 22
MRODout Interrupts
• IRQ0 S-Link Return Lines (LRL) ChangeS-Link LDOWN
• IRQ1 Event Counter Reset (ECR)
• IRQ2 Event/Bunch-ID FIFO FullTrigger Type FIFO Full
All interrupt sources are individually mask-able
Peter JansweijerMROD Design Review: November 12, 2003Slide 23
MRODout FPGA Registers• VMEbus IRQ
– IRQ Level
– 8-bit Status-ID Pattern
• VMEbus BAR and IRQ– Trigger VMEbus IRQ
– IRQ Pending Status
– BAR
• S-LINK Status and Interrupt Register– Link Return Lines (LRL)
– LRL Change Interrupt
– Link Down Interrupt
• TTC Control/Status and Interrupt Register– Event-ID/Bunch-ID/Trigger Type FIFO Empty/Full status
– Event-ID/Bunch-ID/Trigger Type FIFO Full Interrupt
– FIFO Flush select (Software or ECR)
• Resets and LEDs
Peter JansweijerMROD Design Review: November 12, 2003Slide 24
• Event building of data from 3 (/4) MRODin boards
• Formatting of the output data for the ROL
• Testable by SHARC via access to register
Summary MRODout
Peter JansweijerMROD Design Review: November 12, 2003Slide 25
Questions?
MRODin
MRODout
Peter JansweijerMROD Design Review: November 12, 2003Slide 26
Peter JansweijerMROD Design Review: November 12, 2003Slide 27
MROD-1 Reset Topology