Modular Data Acquisition Introduction and applicability to LCLS DAQ

Department of Particle & Particle AstrophysicsDepartment of Particle & Particle Astrophysics

Modular Data Acquisition Modular Data Acquisition

Introduction and applicability to LCLS DAQ

Michael Huffer, [email protected] Linear Accelerator Center

December 14, 2006Representing:Ryan Herbst

Chris O’GradyAmedeo Perazzo

Leonid SapozhnikovEric Siskind

Dave TarkingtonMatt Weaver

Department of Particle & Particle Astrophysics

2


OutlineOutline

• Introduction– Concepts– Architecture– Implementation

• Examples…– Petabyte scale, low access latency storage for SLAC Computer Center– LSST camera data acquisition system

• Application design• Discuss applicability for LCLS Data Acquisition?


3


The ModuleThe Module

• Is the basic building block of the architecture• Specified as:

– A hardware design (schematics, BOM & layout guidelines)– A series of base services implemented as:

• VHDL (interfaced through core IP libraries)• Software (OO interface - provided through header files and shared libraries)

– documentation • Module neither specifies or constrains application’s physical partitioning model• Architecture specifies three different types of modules

– CEM (Cluster Element Module)• Provides a processor + RTOS (the Cluster Element)• Provides many channels of generic, high speed, serial I/O• Provides commodity network interface (10 GE & 100-Base-T Ethernet)

– fCIM (Fast Cluster Interconnect Module)• Provides 10 GE connectivity for up to 64 Cluster Elements

– sCIM (Slow Cluster Interconnect Module)• Provides 100 Base-T & 1 GE connectivity for up to 64 Cluster Elements


4


Cluster Element Module (CEM)Cluster Element Module (CEM)

• Two variants…• footprint:

– ~ 50 cm2

• power:– ~ 7 watts total + – ~ 3/4 Watt/port

JTAGreset

optionsreset

PHYs (4-20)

10 GE 100B-T

CE

To fCIM

To sCIM

A Cluster Element (CE) is a

processor

Each lane operates

up to 10 Gb/sec

CEM (1 channel)

resetoptions

JTAG reset 100B-T10 GE 10 GE 100B-T

PHYs (0-16)

Common to both elements

CECE

May mix and match lanesto each CE

CEM (2 channel)

• Two variants…– One channel CE– Two channel C2


5


FastFast Cluster Interconnect Module ( Cluster Interconnect Module (ffCIM)CIM)

• footprint:– ~ 144 cm2

• Power:– ~ 1 ½ Watt/port– 64 elements ~ 110 watts

1 GE10 GE (0 – 8)

fCIM

Supports a variety of electromechanical

standardsX2 & XENPACK MSA

CX4Long haul & short haul

fibers

To CEIs a collection

of managed switches

To management

network

10 GE (0 – 8)


6


SlowSlow Cluster Interconnect Module ( Cluster Interconnect Module (ssCIM)CIM)

100B-T (2 – 64)

sCIM

1 GE

Supports a variety of electromechanical

standards

To management

or control network

To CE

Is a collection of unmanaged

switches

• footprint:– TBD (less then fCIM)

• Power:– TBD (much less then fCIM)


7


32 32 ElementElement Cluster Cluster

To management

or control network

sCIM

To data network

1 GE

CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE

fCIM

10 GE

fCIM is managed

control network


8


CEM block diagramCEM block diagram

Left side MFD

FPGA (SOC)200 DSPs

Lots of gatesXilinx XC4VFX60

Fabric clock

Right sideMGT clock

Right sidePPC-405

(450 MHZ)

Right sideConfiguration

memory128 Mbytes)

Samsung K9F5608

Right sideMemory

(512 Mbytes)Micron RLDRAM II

Right sideMulti-Gigabit Transceivers

(MGT)8 lanes

Left side 100-baseT

Reset

Reset optionsJTAG


9


Base services provided by CEMBase services provided by CEM

• “Fat” Memory Subsystem– 512 Mbytes of RAM– Sustains 8 Gbytes/sec– “Plug-In” DMA interface (PIC)

• Designed as a set of IP cores • Designed to work in conjunction with MGT and protocol cores

• Bootstrap loader (with up to 16 boot options and images)• Interface to configuration memory• Open Source R/T kernel (RTEMS)• 10 GE Ethernet interface• 100 base-T Ethernet interface• Full network stack • Utility software to manage I/O


10


Extended services provided by CEMExtended services provided by CEM• Pretty Good Protocol (PGP)

– Physical interface is serial with 2 LVDS pairs/lane)– Point-to-Point connectivity– Allows clock recovery– Full duplex

• Symmetric capabilities in either direction from either end– Provides reliable frame (packet) transmission and reception– Deterministic (and small) latency

• Lightweight “on the wire” overhead• Specifies 4 VCs in order to provide QOS

– Implemented as an IP core• Small footprint• Interface hides user from protocol details and implementation• Implemented on CE (through the conical model described above)

– Asynchronous• Extensible in both bit-rate and # of lanes

• Flash Memory Module (FSM)– Provides as much as 256 Bytes/CE of persistent storage– Low latency/high bandwidth access(1 Gbyte/sec)– Interfaced using PGP


11


Cluster Element Cluster Element as used inas used in petacache petacache

FSM FSM FSM FSM

CE10 GE

To/From fCIM

100B-T

To/From sCIM From management

network

PGP 1 lane @

250 Mbytes/se

c

To client nodes on

client network

Called a SAM(Storage Access Module)

65 Gbyte flash memory(Flash Storage Module)

PGP core & interface

Application specific


12


Cluster Element Cluster Element as used inas used in LSST DAQ LSST DAQ

CE10 GE

To/From fCIM

100B-T

To/From sCIM

From Camera Control System on CCS network

To client nodes on DAQ

network

Called a RNA(Raft Network Adapter)

Services 9 CCD mosaic288 Mbytes/sec

Application specific

Raft Readout System

PGP (fiber-300 M) 1 lane @ 300 Mbytes/sec

PGP core & interface

In cryostat

(Replicated 25 times)


13


The “Chassis”The “Chassis”

AcceptsDC power

PassiveBackplan

e

8 U

X2(XENPACK MSA)

1UFan-Tray

1UAir-Outlet

1UAir-Inlet

High-Speed Network Card (8U)

Daughter board Card (4U)


14


Chassis Physical Interfaces (19”)Chassis Physical Interfaces (19”)

CCS network card1 GE

client network card10 GE to odd raft

to even raft

to CCS

to DAQ network

Science Array (12)Guider Array (2)

WFSArray (2)

8U

Number is

TBD

bank

Daughter cards replicated twice for:Redundancy &

simulation


15


• Partition problem into three domains:– Device/sensor specific Read-Out (RO)– Device/sensor monitoring and configuration– Data transport and processing

• Define a consistent and regular interface between RO & CE systems– independent of device/sensor

• Define CE customization– How many lanes of I/O necessary between RO and CE?– What are the protocols on these lanes?– Specify data processing

• How should this processing be partitioned between software and hardware?• CE number

– What is the underlying, inherent, parallelism of the data (if any)?– How many CPU cycles and gates should be dedicated per data byte?

• processing effort/byte• Define physical partitioning of design

– How many boards?– What type and number of modules on a board?– Incorporate with custom logic?

A prescription for application designA prescription for application design

The later two are within the realm of the CE


16


Typical usage patterns Typical usage patterns

CE

RO RO RO RO

RO

CE CE CE CE

RO RO RO

CE CE CE CE

RO

• Many different types of devices• Physically separated• Processing/byte/device is high

• Homogeneous devices• Perhaps physically separated• Processing/byte is high

• Many different types of devices• Physically separated• Processing/byte/device is low

Modular Data Acquisition Introduction and applicability to LCLS DAQ

Documents