The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

The ANTARES Offshore Data Acquisition

A Highly Distributed, Embedded and COTS-based System

S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA

A Neutrino Telescope

http://antares.in2p3.fr

The "0.1 km2" Project

13 Strings30 Detecting Nodes Per String (every 10m)3 Optical Modules (PMs) Per Detecting Node2 "ARS" Digitizing Chips Per Optical Module

Offshore Data Acquisition System: Readout in Real Time 2340 ARS Chips Spread Out Over 30000000m3 @ 2000m UnderwaterOnshore: Perform Real Time Trigger Computations On Incoming Data Farm of 100 PC-Linux Workstations

Detector Topology

ToShore

String ControlModule (SCM)

Local ControlModule (LCM)

Junction box (JB)

Electro Mechanical cable (EMC)

Electro Optical Cable (EOC)

Optical Module(OM)

Slow-ControlDataClockEnergy

In Each LCM: One DAQ Board

ARS 1

Séquenceur

ArbitrageBus MPC860

DAQMemory

Controller

InterfaceBus MPC860

Données

Adresses

COTSProcessorMPC860

max 80MHz

Max P:~700 mW

Integrated:4 Serial

1 Ethernet

1 0 1

DAQMemory

Event Type 1 Event type 2 Event type 3 System Communications

Conversionserial //

ProcessUnit

Data reduction

Time reordering

CRC

MémoireARS

ContrôleMémoireLecture

ARS 2ARS 3ARS 4ARS 5ARS 6

RARS

1 1 0

Ethernet

ARSMemory

Supervisor Control &Status

Register

WriteMemory

controller

ARSMemory

ReadOutMemory

controller

MPC860Bus

Manager

MPC860 BusInterface

Addresses

Data

DAQ board

LCMSlow-Control

System Memory Status

ARS Memory Status

Self-

test

Network Topology (1)

Cable rigidityCable diameter

Connection complexity

SCM

Each

LC

M h

as it

s ow

n I/O

dat

a flo

w(S

low

Con

trol

& P

hysi

cs D

ata)

Sectors

S

S

S

S

M

S

S

SCM

S

S

M

To Shore(DWDM)

Used as data concentratormany input ports one output port

Network Topology (2)

PC PC PC PC PC PC PC

ON-SHORE SWITCH: (in: 78x1000, out: 100x1000)

SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000)

LCM LCM LCM LCM LCM LCM LCM LCM LCM LCM

risks of congestion intelligent flow control in LCM

~15 Mb/s

Massively Distributed System

Final System: A Network of ~500 Nodes– ~400 Offshore Nodes (MPC860 + RTOS)– ~100 Onshore Nodes (PC + Linux)

3 Communicating Parallel Applications– DAQ– Trigger– Slow Control

Software Technologies

Programming Languages– C++ For Fast Software Components– Java for Non Real-Time (not yet mature for RT)

Design & Development– Object Paradigm– UML (Unified Modeling Language) + Extensions– Trigger/DAQ Design Patterns (Functional/Distribution

Separation of Concerns)– Specific Development Methodology for Distributed Systems

(MORDICUS)

Distribution Design Patterns

LCM LCM LCM LCM LCM LCM LCM LCM LCM LCM

PC PC PC PC PC PC PC

CASCADED SWITCHES

Performance Parallelism(Scalability)

Intrinsic Parallelism(Progressive deployment)

acq(in frameID:int)

accept(in f:Frame)

:ARS_Read :Storage

[TRIG = OK]

Sequence Diagram (Functional)

:FrameFormatter

[TRIG = OK]

dump()

:Trigger:EventBuilder

store()

[TRIG OK]

trig(e:Event)

Class Diagram (Functional)

Trigger

+trig(in d:Event) : bool

1

Frame Event

+rawParts0..n

ARS_Read

+fillBuffers(id:int)

0..1+processor

+builder

1

n

+storage

Data

1

+formatter

1

6

+accept(in x: Frame)#dump(in x:Frame)

EventBuilder

FrameFormatter

+acq(in frameID:int)

Storage

+store(in d:Event)

<<fpga>>ALTERA20K

readout

Interrupt - SharedMem

12

PowerPC-RTOS

framer

<<farm>>PC-Linux

processor

<<impl>> <<impl>> <<impl>> <<impl>>

CORBA

**

TriggerFrameFormatter StorageARS_Read

Specialized Deployment Diagram

EventBuilder

<<impl>>

1

+builder

+acq(in frameID:int)

FrameFormatter<<interface>>EventBuilderInterface

+accept(in x:Frame)

Adapts FrameFormatter EventBuilder (Remote + Farm) Communication on Caller Side

Implements Farm Management POLICY

Automatic Model Transformation

1 <<proxy>>EventBuilderProxy

+accept(in x:Frame)

<<farm_manager>>EventBuilder

+accept(in x:Frame)

1 1

New Challenge in HEPReal-Time Computing

Massively Distributed– 400 Offshore Processors– 100 Onshore Processors

Deeply Embedded– Heat Dissipation– Limited Space– Limited Power

Numerous Modules: MTBF Problem– 400 "Satellites" With 1/10th of Budget– Robustness is CriticalDESIGN &

DEVELOPMENT

METHODOLOGY