The ANTARES Offshore Data Acquisition
A Highly Distributed, Embedded and COTS-based System
S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA
A Neutrino Telescope
http://antares.in2p3.fr
The "0.1 km2" Project
13 Strings30 Detecting Nodes Per String (every 10m)3 Optical Modules (PMs) Per Detecting Node2 "ARS" Digitizing Chips Per Optical Module
Offshore Data Acquisition System: Readout in Real Time 2340 ARS Chips Spread Out Over 30000000m3 @ 2000m UnderwaterOnshore: Perform Real Time Trigger Computations On Incoming Data Farm of 100 PC-Linux Workstations
Detector Topology
ToShore
String ControlModule (SCM)
Local ControlModule (LCM)
Junction box (JB)
Electro Mechanical cable (EMC)
Electro Optical Cable (EOC)
Optical Module(OM)
Slow-ControlDataClockEnergy
In Each LCM: One DAQ Board
ARS 1
Séquenceur
ArbitrageBus MPC860
DAQMemory
Controller
InterfaceBus MPC860
Données
Adresses
COTSProcessorMPC860
max 80MHz
Max P:~700 mW
Integrated:4 Serial
1 Ethernet
1 0 1
DAQMemory
Event Type 1 Event type 2 Event type 3 System Communications
Conversionserial //
ProcessUnit
Data reduction
Time reordering
CRC
MémoireARS
ContrôleMémoireLecture
ARS 2ARS 3ARS 4ARS 5ARS 6
RARS
1 1 0
Ethernet
ARSMemory
Supervisor Control &Status
Register
WriteMemory
controller
ARSMemory
ReadOutMemory
controller
MPC860Bus
Manager
MPC860 BusInterface
Addresses
Data
DAQ board
LCMSlow-Control
System Memory Status
ARS Memory Status
Self-
test
Network Topology (1)
Cable rigidityCable diameter
Connection complexity
SCM
Each
LC
M h
as it
s ow
n I/O
dat
a flo
w(S
low
Con
trol
& P
hysi
cs D
ata)
Sectors
S
S
S
S
M
S
S
SCM
S
S
M
To Shore(DWDM)
Used as data concentratormany input ports one output port
Network Topology (2)
PC PC PC PC PC PC PC
ON-SHORE SWITCH: (in: 78x1000, out: 100x1000)
SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000)
LCM LCM LCM LCM LCM LCM LCM LCM LCM LCM
risks of congestion intelligent flow control in LCM
~15 Mb/s
Massively Distributed System
Final System: A Network of ~500 Nodes– ~400 Offshore Nodes (MPC860 + RTOS)– ~100 Onshore Nodes (PC + Linux)
3 Communicating Parallel Applications– DAQ– Trigger– Slow Control
Software Technologies
Programming Languages– C++ For Fast Software Components– Java for Non Real-Time (not yet mature for RT)
Design & Development– Object Paradigm– UML (Unified Modeling Language) + Extensions– Trigger/DAQ Design Patterns (Functional/Distribution
Separation of Concerns)– Specific Development Methodology for Distributed Systems
(MORDICUS)
Distribution Design Patterns
LCM LCM LCM LCM LCM LCM LCM LCM LCM LCM
PC PC PC PC PC PC PC
CASCADED SWITCHES
Performance Parallelism(Scalability)
Intrinsic Parallelism(Progressive deployment)
acq(in frameID:int)
accept(in f:Frame)
:ARS_Read :Storage
[TRIG = OK]
Sequence Diagram (Functional)
:FrameFormatter
[TRIG = OK]
dump()
:Trigger:EventBuilder
store()
[TRIG OK]
trig(e:Event)
Class Diagram (Functional)
Trigger
+trig(in d:Event) : bool
1
Frame Event
+rawParts0..n
ARS_Read
+fillBuffers(id:int)
0..1+processor
+builder
1
n
+storage
Data
1
+formatter
1
6
+accept(in x: Frame)#dump(in x:Frame)
EventBuilder
FrameFormatter
+acq(in frameID:int)
Storage
+store(in d:Event)
<<fpga>>ALTERA20K
readout
Interrupt - SharedMem
12
PowerPC-RTOS
framer
<<farm>>PC-Linux
processor
<<impl>> <<impl>> <<impl>> <<impl>>
CORBA
**
TriggerFrameFormatter StorageARS_Read
Specialized Deployment Diagram
EventBuilder
<<impl>>
1
+builder
+acq(in frameID:int)
FrameFormatter<<interface>>EventBuilderInterface
+accept(in x:Frame)
Adapts FrameFormatter EventBuilder (Remote + Farm) Communication on Caller Side
Implements Farm Management POLICY
Automatic Model Transformation
1 <<proxy>>EventBuilderProxy
+accept(in x:Frame)
<<farm_manager>>EventBuilder
+accept(in x:Frame)
1 1
New Challenge in HEPReal-Time Computing
Massively Distributed– 400 Offshore Processors– 100 Onshore Processors
Deeply Embedded– Heat Dissipation– Limited Space– Limited Power
Numerous Modules: MTBF Problem– 400 "Satellites" With 1/10th of Budget– Robustness is CriticalDESIGN &
DEVELOPMENT
METHODOLOGY