Top Banner
The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar , H. Le Provost, F. Louis – CEA Saclay DAPNIA
15

The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Mar 27, 2015

Download

Documents

Taylor Rollins
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

The ANTARES Offshore Data Acquisition

A Highly Distributed, Embedded and COTS-based System

S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA

Page 2: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

A Neutrino Telescope

http://antares.in2p3.fr

Page 3: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

The "0.1 km2" Project

13 Strings30 Detecting Nodes Per String (every 10m)3 Optical Modules (PMs) Per Detecting Node2 "ARS" Digitizing Chips Per Optical Module

Offshore Data Acquisition System: Readout in Real Time 2340 ARS Chips Spread Out Over 30000000m3 @ 2000m UnderwaterOnshore: Perform Real Time Trigger Computations On Incoming Data Farm of 100 PC-Linux Workstations

Page 4: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Detector Topology

ToShore

String ControlModule (SCM)

Local ControlModule (LCM)

Junction box (JB)

Electro Mechanical cable (EMC)

Electro Optical Cable (EOC)

Optical Module(OM)

Slow-ControlDataClockEnergy

Page 5: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

In Each LCM: One DAQ Board

ARS 1

Séquenceur

ArbitrageBus MPC860

DAQMemory

Controller

InterfaceBus MPC860

Données

Adresses

COTSProcessorMPC860

max 80MHz

Max P:~700 mW

Integrated:4 Serial

1 Ethernet

1 0 1

DAQMemory

Event Type 1 Event type 2 Event type 3 System Communications

Conversionserial //

ProcessUnit

Data reduction

Time reordering

CRC

MémoireARS

ContrôleMémoireLecture

ARS 2ARS 3ARS 4ARS 5ARS 6

RARS

1 1 0

Ethernet

ARSMemory

Supervisor Control &Status

Register

WriteMemory

controller

ARSMemory

ReadOutMemory

controller

MPC860Bus

Manager

MPC860 BusInterface

Addresses

Data

DAQ board

LCMSlow-Control

System Memory Status

ARS Memory Status

Self-

test

Page 6: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Network Topology (1)

Cable rigidityCable diameter

Connection complexity

SCM

Each

LC

M h

as it

s ow

n I/O

dat

a flo

w(S

low

Con

trol

& P

hysi

cs D

ata)

Sectors

S

S

S

S

M

S

S

SCM

S

S

M

To Shore(DWDM)

Page 7: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Used as data concentratormany input ports one output port

Network Topology (2)

PC PC PC PC PC PC PC

ON-SHORE SWITCH: (in: 78x1000, out: 100x1000)

SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000)

LCM LCM LCM LCM LCM LCM LCM LCM LCM LCM

risks of congestion intelligent flow control in LCM

~15 Mb/s

Page 8: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Massively Distributed System

Final System: A Network of ~500 Nodes– ~400 Offshore Nodes (MPC860 + RTOS)– ~100 Onshore Nodes (PC + Linux)

3 Communicating Parallel Applications– DAQ– Trigger– Slow Control

Page 9: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Software Technologies

Programming Languages– C++ For Fast Software Components– Java for Non Real-Time (not yet mature for RT)

Design & Development– Object Paradigm– UML (Unified Modeling Language) + Extensions– Trigger/DAQ Design Patterns (Functional/Distribution

Separation of Concerns)– Specific Development Methodology for Distributed Systems

(MORDICUS)

Page 10: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Distribution Design Patterns

LCM LCM LCM LCM LCM LCM LCM LCM LCM LCM

PC PC PC PC PC PC PC

CASCADED SWITCHES

Performance Parallelism(Scalability)

Intrinsic Parallelism(Progressive deployment)

Page 11: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

acq(in frameID:int)

accept(in f:Frame)

:ARS_Read :Storage

[TRIG = OK]

Sequence Diagram (Functional)

:FrameFormatter

[TRIG = OK]

dump()

:Trigger:EventBuilder

store()

[TRIG OK]

trig(e:Event)

Page 12: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

Class Diagram (Functional)

Trigger

+trig(in d:Event) : bool

1

Frame Event

+rawParts0..n

ARS_Read

+fillBuffers(id:int)

0..1+processor

+builder

1

n

+storage

Data

1

+formatter

1

6

+accept(in x: Frame)#dump(in x:Frame)

EventBuilder

FrameFormatter

+acq(in frameID:int)

Storage

+store(in d:Event)

Page 13: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

<<fpga>>ALTERA20K

readout

Interrupt - SharedMem

12

PowerPC-RTOS

framer

<<farm>>PC-Linux

processor

<<impl>> <<impl>> <<impl>> <<impl>>

CORBA

**

TriggerFrameFormatter StorageARS_Read

Specialized Deployment Diagram

EventBuilder

<<impl>>

Page 14: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

1

+builder

+acq(in frameID:int)

FrameFormatter<<interface>>EventBuilderInterface

+accept(in x:Frame)

Adapts FrameFormatter EventBuilder (Remote + Farm) Communication on Caller Side

Implements Farm Management POLICY

Automatic Model Transformation

1 <<proxy>>EventBuilderProxy

+accept(in x:Frame)

<<farm_manager>>EventBuilder

+accept(in x:Frame)

1 1

Page 15: The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA.

New Challenge in HEPReal-Time Computing

Massively Distributed– 400 Offshore Processors– 100 Onshore Processors

Deeply Embedded– Heat Dissipation– Limited Space– Limited Power

Numerous Modules: MTBF Problem– 400 "Satellites" With 1/10th of Budget– Robustness is CriticalDESIGN &

DEVELOPMENT

METHODOLOGY