HSLB board User Guide
Belle II note XXX
Belle2link Design
Ver. 0.1
Date: Feb. 25 2011
By Zhen-An Liu, Dehui Sun, Jingzhou Zhao, IHEP
Note:
This document will be a full description of Belle2link system to
be used in Belle II experiment. This file also functions a guidance
in early stage to the subsystems in designing the interfacing to
Belle2link in Front-End electronics.
But still it is in the very basic stage, we are sorry about
this. You suggestions and comments are welcome.
0. Table of Contents
i. Introduction
ii. BL_RSVR,FINESSE card
iii. BL_SNDR
iv. Demo system for CDC
v. supplements
Introduction
The Belle II collaboration has decided to use optical fibers and
RocketIO to make the connection between Front-End Electronics
(FEE)/Trigger system and DAQ system for signal and data
transmission. For system simplicity and reliability, a so-called
Belle2Link, an unified high speed link system, has been defined for
use in overall connection between DAQ and each of the subsystems.
It features as:
1. unification in hardware design
2. unification in firmware design
3. provides electrical isolation
4. provides high speed transmission rate
5. versatile for different input data rate
6. home brew transmission protocol
Figure 1-1 the relation of Belle2Link with FEE and DAQ
As Belle II DAQ will based on the COPPER structure, Belle2Link
comprise mainly RocketIO and optical fiber based high speed
transmission lines and interfaces to FEE(IF FE), to global trigger
and timing distribution(IF TTC) and to PCI bridge on COPPER (IF CO
)(Figure 1-1), i.d. it accepts data in FIFO FE via IF FE from FEE
for transmission in the sender part (BLSNDR), gets trigger and
timing signals from timing and trigger system via IF TTC, and at
the receiver part (BLRSVR) save the received data in the FIFO BL
and moves data after necessary processing to the FIFO CO in IF CO
which is actually on the COPPER(Figure 1-2).
As a unified link for all sub-detector system, Belle2Link
provides same or limited type of receiver FINESSE boards, necessary
merger board, and all the unified firmware for sender part and
receiver part.
Figure 1-2 Block diagram of the Belle2Link
1.1 Unification in hardware
Not as a separate part, the BLSNDR will rather share the
resources of and be implemented in a Virtex-5 FPGA in the most FEE
readout board. The FEE readout of each sub detector systems provide
the V5 GPGA and the BLSNDR part of the unified Belle2Link firmware
will be integrated with the readout firmware. The requirement to
the FEE from Belle2link includes:
a. An SFF optical transceiver connected to FPGA GTPs
b. On board crystal of 125MHz
c. RJ-45 connector receiving system clock
For some system where necessary a merger is added before the
BLSNDR. The high precision clock CLK LK for the high speed
transmission via RocketIO will be same for all Belle2Link. This CLK
LK is preliminarily defined as 125MHz to get a line rate of
3.125Gbps and be produced by a high precision crystal both on the
FEE board and FINESSE receiver board(BLRB). BLRD board is FINESSE
board which is fit the general requirement of high speed
transmission and also some data processing required and fits to the
COPPER board electrically and mechanically.
1.2 Unification in firmware and protocol
Most of the circuitry of the Belle2Link will be implemented by
firmware in the Xilinx FPGA. The development of the link will be
based on Xilinx ISE 11 which supports the Virtex-5 and Spartan-6.
The firmware for BLSNDR will be unified and provided to all system
for integration to the system. IHEP group will provide as an
example the full firmware for CDC subsystem which can be directly
used in other system as long as the version regulation is followed.
The working status can be checked by registers and displayed on
some LED on the front panel. The interface with FEE is a
FIFO(FIFO-FE). The FIFO FE is a 32bit write and 16bit read FIFO and
will be filled with data by FEE readout under control of clock CLK
FE and then be moved to GTP under sender clock CLK LN after adding
some control and error checking data by BLSNDR state machine. Some
of the slow control like function has been added for parameter
setting and status/error checking.
In the BLRSVR, another state machine checks for the GTP line .
The data on the line will be first received and stored in a FIFO-LN
under control of clock CLK LN. If it is error free this data will
be moved to a FIFO LN (N EV will be incremented). If there is a
error occurred on the transmission, an error flag is added in the
header part of data frame for later use in the online data
monitoring. The data will then be check again normal data or slow
control data. If it is slow control data it will be send back to
the requesting task. Otherwise the data in FIFO-LN will be direct
moved to another fifo FIFO WR, but for systems like B-PID the date
must be first moved to a pipeline(Figure ??), and part of the data
in a time window will be fetch after receives a trigger and be
processed accordingly, then be stored in FIGO WR in the data form
showed in table ??. A special error bits has been allocated which
indicates if there is a data transmission error occurred.
table ??
Another state machine(Figure ??) checks the NEV WR, if there is
an event in the FIFO WR, then send the data for this event to FIFO
CO under control of clock CLK CO and decrease the NEV WR if the
COPPER is ready(No FFUL signal) to read. At the same time if the
number of event NEV WR exceeds a predefined number a busy signal is
produced for TTRX usage in blocking the further triggers and be
removed two CLK CO clocks after data has been moved to FIFO CO. The
error checking scheme will be kept for the online checking of the
transmission between Belle2Link and COPPER.
1.3 Clocks
Three key clocks will be needed by Belle2Link, CLK FE which is
converted from central trigger and timing distribution, CLK LK for
the high speed transmission which is a high precision clock from a
crystal on the board. CLK_LK is connected to on board crystal as
default, and then switched to the RJ-45 system clock later in
normal running state. CLK CO is received from TTRX via COPPER
connector. The clocks for data transmission on the RocketIO will be
composed by the DCS in Virtex-5 core. For the prototype system, we
have used a CLK LK of 200MHz and get successfully a line rate of
2Gbps.
2. BL_RSVR: receiver card
A receiver FINESSE card, or High Speed Link Board( HSLB) has
been designed for use for a high speed data receiving on COPPER
board. It uses the GTP and optical fiber to realize the Gigabit
speed transmission. The rate can reach up to 3.125Gbps. Figure 2-1
shows the block diagram of HSLB board.
H
S
L
B
I
N
T
E
R
F
A
C
E
FPGA
XC5VLX50T
FLASH
XCF16P
CPLD
XC2C64A
Optical transceiver
Optical transceiver
RJ45
RocketIO
Clock
Control Signal
Data Bus
Address Bus
Trigger Signal
Figure 2-1 block diagram of HSLB board.
2.1 Key Components and Features
Figure 2-2 shows the HSLB board, which includes the following
components and features:
XCLX50T-1FF1136C FPGA
2x SFP module ,
LED for SFP which can be redefined by user.
LED for indicate the status of the SFP
RJ45 port
LVDS OSC for debugging the board
MC100LVEP14DTR2 for distributing the system clock
MC100EPT21D for transforming the LVDS Clock to single Clock
signal.
200M LVDS OSC for GTP
16Mbit Xilinx XCF16PVOG48C Platform Flash, in-system
programmable configuration PROM
Xilinx XC2C64A-7VQG100C CPLD for controlling and configuring the
PROM and FPGA
JTAG interface
HSLB board to COPPER board interface A
HSLB board to COPPER board interface B
HSLB board to COPPER board interface C
Eight individual LED
Eight individual test pins and ground pins
P/N signal test pins
3.3V Power-on indicator LED and FPGA configuration DONE LED
On-board 2.5Vregulators. , 1.2VTT , 1.2VPLL, 1.0VMGT, 1.0VINT,
1.8V
Fuze of 3.3V and 5V
27
13
12
33
11
10
987
6
5
4
3
2
1
14
16
15
17
19
18
29
30
31
28
26
25
20
21
22
23
24
12
14
13
32
Figure 2-2 shows the component location.
2.2 SFP interface and LED
HSLB board uses the 112 GTP dual(X0Y3) and 114 GTP dual(X0Y2)
for SFP. The table 2-1 shows the SFP and SFP LED control
signals.
112GTP (X0Y3)
114 GTP (X0Y2)
Signal
Pin#
Signal
Pin#
TXFAULT
J6
TXFAULT
N8
TXDISABLE
E6
TXDISABLE
K6
LOS
G5
LOS
N5
MOD0
F6
MOD0
M6
MOD1
E7
MOD1
L5
MOD2
F5
MOD2
L4
RATESEL
J5
RATESEL
N7
LED0
E8
LED2
K8
LED1
F8
LED3
K9
2.3 RJ45 Port
The board can bring in external clock signal, trigger signal and
other control signal though RJ45. Table 2-2 shows the pin
distribution.
RJ45 signal distribution
Signal
RJ45 Pin#
FPGA Pin#
SYSCLK_N
P1
AD5
SYSCLK_P
P2
AD4
INI_N
P3
AK7
INI_P
P6
AK6
RSVCLK_N
P4
AC7
RSVCLK_P
P5
AD7
SYSDAT_N
P7
AE6
SYSDAT_P
P8
AD6
LED1
P9
AC4
LED2
P12
AB6
2.4 LED and Test Pins
The HSLB board has eight individual LEDs and test pins. They are
directly connected to the FPGA Pin. Table 2-3 shows the pin
distribution.
Test pin P2,P4,P6,P8,P10,P12,P14,P16 are joined to the
ground.
LED
FPGA Pin#
TEST Signal
FPGA Pin#
DS705
D10
P1
AM11
DS706
D11
P3
AK11
DS707
B12
P5
AL11
DS708
C12
P7
AL10
DS709
D12
P9
AJ11
DS710
A13
P11
AJ10
DS711
B13
P13
AH10
DS712
C13
P15
AG10
2.5 Clock Sources
HSLB board has four LVDS clock signals.
The clock comes form the HSLB to COPPER interface C, SCK+/-. It
is used for the system clock, and synchronous with the copper read
and write HSLB clock.
The clock comes form the RJ45, the external clock SYSCLK_N/P.Pin
distribution see table 2-1.
The OSC clock on the back of the board is high accuracy. It is
used for the GPT.
The clock on board, is used for testing clock.
Table 2-4 shows the FPGA pin distribution for clock signal.
GTP clock
P
P4
N
P3
Testing clock
P
AH20
N
AH19
System clock
P
AG22
N
AH22
2.6 JTAG Programme
The HSLB board includes a JTAG programming and debugging chain.
All the Virtex-5 FPGA, the Platform Flash and CPLD devices are part
of the JTAG chain, as shown in Figure 2-3.
XCLX50TFLASH
6
8
4
10
TMS
TDI
TDO
TCK
TDI
TDI
TDO
TDO
TMS
TMS
TCK
TCK
J1
CPLD
TDI
TD0
TMS
TCK
2.7 Platform Flash Configuration Storage
The HSLB board has an XCF16PVOG48C configuration Flash PROM to
store FPGA configuration data. The Flash provide serial and
parallel configuration. And the three signal M0,M1,M2 which decide
the configuration should be assigned in the CPLD. Figure 2-4 show
the way Flash connect to FPGA and CPLD.
FLASH
XCF16P
CPLD
XC2C64A
FPGA
XC5VLX50T
Address Bus
Data Bus C_LD[0:7]
C_LA[0:6]
C_CCLK
C_CSB
C_INIT_B
C_BUSY
C_CE_B
C_LWR
C_RSV[0:3]
C_M[0:2]
C_RDWR_B
C_PRGM_B
C_DIN
C_DIN
C_BUSY
C_CS_B
C_INIT_B
2.8 CPLD
CPLD is used as the interface chip in the HSLB board. Software
can be download to CPLD, then the system can configure the FPGA and
Flash online. Table 2-5 show the detail signals and the pin
distribution of the CPLD.
Signal
CPLD Pin#
Signal
CPLD Pin#
C_LD0
P64
F_LD0
P27
C_LD1
P61
F_LD1
P24
C_LD2
P60
F_LD2
P22
C_LD3
P58
F_LD3
P19
C_LD4
P56
F_LD4
P18
C_LD5
P55
F_LD5
P17
C_LD6
P53
F_LD6
P16
C_LD7
P52
F_LD7
P15
C_LA0
P67
F_LA0
P14
C_LA1
P68
F_LA1
P13
C_LA2
P70
F_LA2
P12
C_LA3
P71
F_LA3
P11
C_LA4
P72
F_LA4
P10
C_LA5
P74
F_LA5
P9
C_LA6
P76
F_LA6
P8
C_INIT_B
P29
C_SCK
P23
C_CCLK
P30
C_PRGM_B
P32
C_DIN
P33
C_DONE
P34
C_CS_B
P35
C_RDWR_B
P36
C_BUSY
P37
C_M0
P39
C_M1
P41
C_M2
P40
C_CE_B
P42
C_CSB
P78
C_LWR
P77
F_LWR
P7
F_CSB
P6
C_RSV0
P92
C_RSV1
P91
C_RSV2
P90
C_RSV3
P89
2.9 Power Distribution
The HSLB board use TPS74401 as the 2.5V, 1.8V, 1.2VPLL, 1.2VTT,
1.0VMGT and 1.0VINT power regulater. TPS74401 is 3.0A Ultra-Low
Dropout Linear Regulator. At the output of every power regulater
circuit, 0R resistance is used to as the isolating resistance.
2.10 HSLB to COPPER interface
On the HSLB there are three socket for connecting COPPER, A is
for transiting the mass data; B is for the power; and C is for
trigger signals, clock signals, and other control signals. Table
2-6 shows detail of the HSLB to COPPER interface signals.
A
B
C
Pin#
Signal
FPGA Pin
Pin#
Signal
Pin#
Signal
FPGA/CPLD Pin
A1
FF00
B32
B1
-3.3V
C1
IRSTB
FPGA_E31
A2
FF01
A33
B2
-3.3V
C2
IENA
FPGA_F30
A3
FF02
B33
B3
-3.3V
C3
IO
FPGA_F31
A4
FF03
C32
B4
-3.3V
C4
TYP0
FPGA_G30
A5
FF04
C33
B5
-3.3V
C5
TYP1
FPGA_G31
A6
FF05
C34
B6
-3.3V
C6
TYP2
FPGA_H30
A7
FF06
D32
B7
GND
C7
TYP3
FPGA_J30
A8
FF07
D34
B8
GND
C8
TAG0
FPGA_J31
A9
FF08
E32
B9
GND
C9
TAG1
FPGA_L30
A10
FF09
E33
B10
GND
C10
TAG2
FPGA_L31
A11
FF10
E34
B11
GND
C11
TAG3
FPGA_M30
A12
FF11
F33
B12
GND
C12
TAG4
FPGA_M31
A13
FF12
F34
B13
+3.3V
C13
TAG5
FPGA_N30
A14
FF13
G32
B14
+3.3V
C14
TAG6
FPGA_P30
A15
FF14
G33
B15
+3.3V
C15
TAG7
FPGA_P31
A16
FF15
H32
B16
+3.3V
C16
LD0
CPLD_P27
A17
FF16
H33
B17
+3.3V
C17
LD1
CPLD_P24
A18
FF17
H34
B18
+3.3V
C18
LD2
CPLD_P22
A19
FF18
J32
B19
GND
C19
LD3
CPLD_P19
A20
FF19
J34
B20
GND
C20
LD4
CPLD_P18
A21
FF20
K32
B21
GND
C21
LD5
CPLD_P17
A22
FF21
K33
B22
GND
C22
LD6
CPLD_P16
A23
FF22
K34
B23
GND
C23
LD7
CPLD_P15
A24
FF23
L33
B24
GND
C24
LA0
CPLD_P14
A25
FF24
L34
B25
+5V
C25
LA1
CPLD_P13
A26
FF25
M32
B26
+5V
C26
LA2
CPLD_P12
A27
FF26
M33
B27
GND
C27
LA3
CPLD_P11
A28
FF27
N32
B28
GND
C28
LA4
CPLD_P10
A29
FF28
N33
B29
-5V
C29
LA5
CPLD_P9
A30
FF29
N34
B30
-5V
C30
LA6
CPLD_P8
A31
FF30
P32
B31
GND
C31
LWR
CPLD_P7
A32
FF31
P34
B32
GND
C32
CSB
CPLD_P6
A33
GATE
R33
B33
+12V
C33
TRG+
FPGA_AJ31
A34
FRSTB
R32
B34
GND
C34
TRG-
FPGA_AK31
A35
FWENB
T34
B35
-12V
C35
REV+
FPGA_AJ30
A36
FWCLK
T33
B36
GND
C36
REV-
FPGA_AH30
A37
FFUL
R34
C37
RCK+
FPGA_AH29
A38
NWFF
U31
C38
RCK-
FPGA_AG30
A39
ABRT
U32
C39
SCK+
to MC100LVEP14DTR2
A40
BSYB
U33
C40
SCK-
to MC100LVEP14DTR2
3. BL_SNDR
3.1 Physical interface at the fronted readout board
Each of the FEE readout board must provide one SFP transceiver
which is linked to GTP_Dual Tx/Rx port of Virtex 5 FPGA as shown in
Fig 3.2.
The design of power supply for GTP must fulfill the requirements
expressed in RocketIO GTP Transceiver part of the datasheet.
Fig.3-1 Peripheral signals at the CDC readout board
3.2 Clocks
For a proper high-speed operation, the GTP transceiver requires
a high-quality, low-jitter, reference clock. Belle2Link need a
special external oscillator which must fulfills the requirements
given in the GTP Transceiver section of the Virtex-5 FPGA Data
Sheet. Two external Clock must be provided on the FEE readout
board.
1. on board oscillator input
Each GTP_DUAL tile has a pair of dedicated pins that can be
connected to the external on board clock source. We recommend using
AC coupling between the oscillator output pins and the dedicated
GTP_DUAL clock input pins.
2. RJ-45 system clock input
We recommend to connect the RJ-45 system clock via another
nearby dedicated GTP_DUAL clock input pins to have good clock
quality. As this clock cannot be used by other parts of FPGA before
embedded PLL in GTP IPcore, RJ-45 clock could be connected via a
global clock pin if you have to.
The internal clock nets of the FPGA provide the reference clock
for the GTP_DUAL tile by connecting the output of a global clock
buffer (BUFG) to the CLKIN port. But this type of clocking, called
GREFCLK clocking, has the lower performance with more jitter.
GREFCLK clocking should be avoided, if possible. See the Virtex-5
FPGA Data Sheet for the jitter margins at different speeds.
Fig.3-2 BL_SNDR Clocking
3.3 Structure of Belle2Link firmware
The BL_SNDR firmware in FEE readout board FPGA has three main
functional components. The slow control module configures or reads
the FEE registers. The data link module receives the data from FEE
and packs them. The protocol module receives the package from GTP
module, check the packages and unpack them. It also deals with the
data transmission for CDC and arbitrate the priority between slow
control module and data link module.
Fig.3-3 Structure of BL_SNDR firmware in CDC readout board
The firmware of BL_RCVR has the same function components. The
protocol module receives the package from GTP module, check the
packages and unpack them, separate the data and transfer them to
slow control module or data link module. Data Link module receive
the data of FEE, transmit the data and the clock to the COPPER
FIFO. Slow Control module receives or transfer the slow control
message from Local bus.
Fig.3-3 Structure of HSLB firmware in CDC readout board
3.3.1 The definition of the interface between FEE and
BL_SNDR
The BL_SNDR should be designed to be transparent to COPPER. So
the interfaces between FEE and BL_SNDR are designed to be similar
as BL_RCVR to COPPER. The interface for FEE data exchange looks
same as the COPPER FIFO Interface (Fig.3-4). The FEE transmit the
data with an synchronous clock signal. I recommend keeping 32-bit
width for FEE data. The existence of an enable signal indicates the
length of an event.
Fig.3-4 Timing chart for Data interface
The interface for slow control, called parameter bus, is similar
to the Local Bus on COPPPER (Fig.3-6). If COPPER writes a value on
address 0x41 of the Local bus, the value will be send to the same
address on parameter bus after a few nanoseconds.
The time charts of Belle2Link slow control at FEE are as follow
(Fig.3-5).
Fig.3-5 timing chart for Parameter bus
3.3.2. Implementation of Slow Control
Fig.3-6 structure of slow control module in BL_RCVR firmware
Fig.3-7 structure of slow control module in BL_SNDR
There are two approaches for the implementation of slow
control.
For the large stream, such as DSP ROM file and configure file,
the system will write the file on a special address 0x00 serially.
The Slow control module processes the stream, and then transfers
the file to protocol module in form of package. The file is
transferred to FEE by the Belle2Link,and written on the same
address 0x00 on parameter bus on FEE board serially. There is no
buffer in Belle2Link. The definition and structure of the file
absolutely depends on the subsystems.
For individual setting (less than 128 bytes), the Belle2Link
uses two dual-port RAM for the reading and writing separately
(Fig.3-6). How to configure the register of FEE is straightforward,
the system writes the value on some address, the values are
recorded in a RAM, the addresses are recorded in a FIFO. Slow
control module organizes the configure message and transfer them to
the protocol module. Because of the latency of the Belle2Link, the
FEE register reading cannot works like configuration. But the
reading is no so straightforward for the system. The COPPER must
read an address twice to complete one reading action. The first
reading informs the Belle2Link which addresses need to be read. The
second reading will get the value which was read from FEE board by
Belle2Link. The reading works like a handshake processing with the
help of a CSR (control and status register).
3.3.3 Arbitration of data sending
For the slow control message and the event of FEE are
transmitted in one line, the protocol module of BL_SNDR watches the
status of two links,arbitrates the priority level and processes the
message and date into different package. There are 3 priority
levels. The state machine of the protocol is as follow
(Fig.3-8).
Fig.3-8 state machine of BL_SNDR protocol module
3.3.4. Processing of data in BL_RCVR
The receiving data are processed separately by different f of
packages based on the package header. The slow control stream is
recovered to the value and address. The normal data stream is
processed to satisfy the format of COPPER. (Fig.3-9)
Fig.3-9 state machine of BL_RCVR protocol module
4. Demo System for CDC
The demo system for CDC is a sample witch realized all the
function of Belle2Link
4.1 Hardware
The demo system consists the following modules (Fig.4-1).
· COPPER board (mother board, 9U size)
· High Speed Link Board (Plugged to COPPER)
· CDC readout board (Linked to HSLBs with optic fibers)
· Server PC (file system server)
Fig. 4-1 Hardware components for CDC setup
4.2 Firmware
HSLB main FPGA firmware ( Xilinx Virtex 5 XC5VLX50t-1ff1136
)
HSLB CPLD firmware (online downloading)
CDC FPGA firmware
4.3 Application Software and HSLB Driver
The Software
Data processing
· Readhslb : read the data of FEE from the COPPER;
Slow control
· paraconf : read or configure the register of HSLB and FEE
· fileconf : transfer file to FEE, such as a DSP ROM file
Driver
Inherit from Higuchi-san’ Finesse general driver
5. Supplements
_1234567890.vsd
�
�
�
�
�
HSLB INTERFACE
FPGA
XC5VLX50T
FLASHXCF16P
CPLD
XC2C64A
Optical transceiver
Optical transceiver
RJ45
RocketIO
Clock
Control Signal
Data Bus
Address Bus
Trigger Signal