OO Software and Data Handling in AMS Computing in High Energy and Nuclear Physics Beijing, September 3-7, 2001 Vitali Choutko, Alexei Klimentov MIT, ETHZ.

OO Software and Data Handling in AMS

Computing in High Energy and Nuclear Physics

Beijing , September 3-7 , 2001

Vitali Choutko, Alexei Klimentov

MIT, ETHZ

A.Klimentov AMS software and data handling CHEP01

2

Outline AMS – particle physics experiment on

the international space station :

Data flow and AMS ground centers Software development Conditions and Tag Database Data Processing

AMS Detector STS91 precursor flight AMS ISS mission


3

AMS : a particle physics experiment in space

PHYSICS GOALS :

Accurate, high statistics measurements of charged, cosmic ray spectra in space > 0.1GV

Nuclei and e- spectra measurement• The study of dark matter (90% ?)

• Determination of the existence or absence of antimatter in the Universe Look for negative nuclei

• The study of the origin and composition of cosmic rays Measure isotopes D, He, Li,

Be…

+


4


5

Magnet : Nd2Fe14BTOF : trigger, velocity and ZSi Tracker : charge sign, rigidity, Z Aerogel Threshold Cerenkov : velocityAnticounters : reject multi particle events

Results :

Anti-matter search : He / He = 1.1x 10Charged Cosmic Ray spectra Pr, D, e- , He, NGeomagnetic effects on CR under/over geomagnetic cutoff components

10 events recordedTrigger rates 0.1-1kHzDAQ lifetime 90%

8

-6_

Precursor flight :

+


6


7

AMS on ISS , 3 years in space

Separate e- from p,p+_

up to 300 GeV

He, He, B, C…

3 4

e-,up to 1000 GeV

+


8

Tracker

TRD

TOF (S1, S2)

MagnetHe Vessel

RICH

ECAL

TOF (S3, S4)

R. BeckerAugust 15, 2001

Veto Counter

USS

Radiators

Electronics

AMS 02 In Cargo Bay

separate e from p,p up to 300 GeV

± −

8Layers

e,to1000GeV±

He,He,B,C,...3

4

AMSonISSfor3years


9

Monitoring& science data

Stored data

Real-timeDataH&S

Real-time & “Dump” data

Real-time, “Dump”, & White Sand’s LOR playback

AMS

ACOP

High Rate Frame MUX

White Sand, NM facility

MSFC, Al

Payload DataService system

Telescience centers

Exte

rnal C

om

mu

nic

ati

on

s

GSELongTerm

ShortTerm

PayloadOperationsControlCenter

ScienceOperationsCenter

ISS to Remote AMS Centers Data Flow

ISSNASAGroundInfrastructure

RemoteAMS Sites

H&SMonitoringScienceFlight ancillarydata

Real-time & “dump”

NearReal-time

File transferplayback

A.Klimentov CHEP01 10

AMS Ground Centers

Science Operations Center

POCCPOCCPOIC@MSFC AL

AMS Remotecenter

RT data CommandingMonitoringNRT Analysis

NRT Data Processing Primary storage Archiving DistributionScience Analysis

MC productionData mirror archiving

Exte

rnal

Com

mu

nic

ati

on

s

ScienceOperationsCenter

XTermHOSC Web Server and xterm

TReK WS

commandsMonitoring, H&S dataFlight Ancillary dataAMS science data (selected)

TReK WS“voice”loop

Video distribution

Production Farm

AnalysisFacilities

PC Farm

Data Server

AnalysisFacilities

GSE D S

A eT rA v e r

GSEBuffer dataRetransmitTo SOC

AMS Station

AMS Station

AMS Station

GSE

MC production

cmds archive

AMS Data, NASA data,

metadata


11

AMS SW development

Been started mid 1996 : basic decisions : new code C++ only (though we had a large part

of legacy SW written on Fortran) Existing libraries (CERNLIB, Geant, etc)

incorporated via C/Fortran interface (R.Burow) transient and persistent classes are separated

with implementing of copy member functions Decide to use Root and HBOOK for histogramming

and data visualization


12

AMS SW development (cont’d) Use different persistency solutions for

various type of data :Flat files for the raw dataNtuples and Root files for ESDRelational Database (Oracle) tables for file cataloguesRelational Database (Oracle) [Objectivity up to Sep 1998]

o Event Tags

o Calibration data

o Slow control data

o NASA ancillary data

o Various catalogues (processing history, etc)


13

Tag Storage with Oracle RDBMS

Tag is an unsigned 32 bit integer containing 16, 1 to 5 bit long parameters such as charge, momentum sign, ß,…

Model :

Query : retrieve tags with 3 parameters satisfied to the given limits (query taken from the “real analysis chain”)

Data stored on Raid array connected to AS4100 (quad-CPU rated at 600MHz, 2GB RAM)

• Flat files – 2400 files, one file per DAQ run, tags are stored as an array of unsigned int.• RootN - 10 files, each file with ~240 trees, one tree per DAQ run with single branch (tag) per tree• RootS - 10 files, each file with ~240 trees, one tree per DAQ run , having 16 branches, every parameter stored in a dedicated branch• OracleN - table with 10 partitions and 1 column, mapping tag to a column• OracleI - table with 10 partitions and 1 column with 16 bitmap indices, mapping tag to a column• OracleS – table with 10 partitions and 16 columns, every parameter mapping to a column


14

Oracle RDBMS to store AMS tags

Method Size Gbyte

Query Time (sec)

Total write time (sec)

Record write time (sec)

Flat Files 1.4 600 - -

RootN 0.9 700 2168 22

RootS 1.2 112 2200 66

OracleN 3.4 1420 6600 66

OracleS 6.6 600 6600 66

OracleI 3.4 3.9 6600+500 661)

1) 500 sec to build indices for 100M tags


15

Design of the Conditions Database Collection of Time Dependent Values (TDVs)

Primary access keys : name, id, validity interval Secondary key : insert time Major Components : table of names and ids, default TDVs,

TDVs Applications : Loading data into database Fetching conditions during event reconstruction Management utilities (TDV browser)

•Name, id•Validity begin, validity end time•Insert time•Array of unsigned integers (size 100 byte – 8 Mbyte)


16

AMS Conditions Database Initially Objectivity, then flat files, now Oracle Performance test for

TDV Total Records

Flat file size (Mbyte)

Oracle table size (Mbyte)

Record write time (msec)

TOF Temperature 9835 1.9 2.8 17

Tracker pedestals (a) 330 36.3 45.2 75

Tracker pedestals (b) 330 36.3 44.9 103(a) BLOB array is stored inside the table, (b) - outside

TOF temperature (many short records)Tracker pedestals (small amount of large records)


17

Oracle RDBMS to store Tags and TDVs

Currently 8 Gbyte is stored in the Conditions DB (115 different TDV types)

100 million event tags are stored in Tag DB Oracle RDBMS performance and functionality

satisfy AMS requirements. Using of bitmap indices for tags improves query time dramatically.

The current implementation works with distributed CORBA technology. It allowes to reduce the number of database clients and machine loading.


18

producers

producers

Raw data

server

server

Oracle RDBMS

Conditions DB

Tag DB

Active Tables :Hosts, Interfaces,

Producers, Servers

Catalogues

server

server

server

server

server

server

Nominal Tables

Hosts, InterfacesProducers, Servers…

ESD

ESD

ESDESD

ESD

server

Raw data

{I}{II}

{III}

{IV}

{V}

{VI}

{VI}

• {I} submit 1st server•{II} “cold” start•{III} read “active” tables (available hosts, number of servers, producers, jobs/host)•{IV} submit servers•{V} get “run”info (runs to be processed, ESD output path)• {VI} submit producers (LILO, LIRO,RIRO…)•Notify servers

AMS Production


19

AMS Production Highlights Stable running for more than 1 month Average efficiency 95% (98% without Oracle) Processes communication and control via Corba LSF for process submission Run Oracle server on AS4100 Alpha and Oracle clients on

Linux. Oracle RDBMS

Tag DB with 100M entries Conditions DB with 100K entries Bookkeeping

Production status Runs history File catalogues

OO Software and Data Handling in AMS Computing in High Energy and Nuclear Physics Beijing, September 3-7, 2001 Vitali Choutko, Alexei Klimentov MIT, ETHZ.

Documents

nasa data

data handling chep01magnet

data visualizationams

various type of data

ethz ams software

gev ams software

oo software

ams computing