INSTR02 1
Computing in High Energy Physics
INSTR02March 6 2002
René Brun
CERN
Rene Brun Computing in HEP / INSTR02 2
Computing in High Energy Physics
A wide subject I want to focus on what I know best
Offline Computing and its relations with Online
Data Storage & Management Software Frameworks Distributed Data Access & Analysis the GRID projects
Show Evolution and discuss trends
Rene Brun Computing in HEP / INSTR02 3
The OLD Model
Data StructuresBanks Managers
ZEBRA BOS
Experimentspecific software
in FortranEvent Generators
Pythia, Isajet
CERNLIB
Geant3 PAW
Rene Brun Computing in HEP / INSTR02 4
Understanding Trends
Looking backward (past 20 years) is necessary to understand the current trends.
I am not reading in the crystal ball. My perception is subjective and biaised by my current involvement in the ROOT project. View is LHC-centric.
I will not risk myself beyond the LHC startup (or more precisely 2006/2007).
Some examples from the ALICE collaboration
Rene Brun Computing in HEP / INSTR02 5
Move to OO Programming The first attempts to introduce the OO concepts in HEP software were
made in 1989. These attempts were unsuccessful for many reasons:
OO languages were not mature and not widely available.
The performance of compilers could not compete with the Fortran compilers.
The introduction of a new system requires that this new system provides all the facilities of the previous system in a stable and proven environment. This is a chicken and the egg problem. Convincing the silent majority to adopt a new tool or framework is a challenging work.
The problem of Object Persistency had been totally underestimated or a wrong solution assumed.
Last, but not least, the introduction of a new system could only be done with the strong support of a major laboratory and a long term commitment. The situation at CERN had been very confusing.
Rene Brun Computing in HEP / INSTR02 6
LanguagesIn 1991, my boss said:
“I do not know what will be the future language
but I know that it will be called Fortran”
Fortran: rapidly vanishing
C++: the main stream language
Java: much less used than expected
C# the Microsoft big ?
Rene Brun Computing in HEP / INSTR02 7
Java: The end ?
Java is the current maintream language for Web-based
applications. Still not really portable. Java applications seen as very slow. MS wants to kill Java.
JAS: Interesting features, but not used because expmts went to C++ and Linux.
Rene Brun Computing in HEP / INSTR02 8
ROOT Downloads
123,000 binariesdownload
1,300,000 clicksper month
35,000 docsin 12 months
2200 reg usersin roottalk
Rene Brun Computing in HEP / INSTR02 9
Productivity
C++ is a very powerful language, but extremely difficult to learn.
Transition from the Fortran world has been very painful.
Slow process because basic libraries not available in the early days
Rene Brun Computing in HEP / INSTR02 10
A bit of History
1994/95 Official line projects start RD44 (Geant4), RD45 (Objectivity)
Jan 1995. ROOT project starts in NA49,ALICE Sep 1995. LHC++ (CERNLIB replacement) starts Oct 1995. JAS (Java Analysis Studio) starts (SLAC) Nov 1998. CDF adopts ROOT, followed by STAR, RHIC,etc Jan 2000. LHC++ renamed to Anaphe Oct 2001. CMS changes its baseline from Objectivity to
ROOT Feb 2002. ROOT is used by most HEP experiments LCG/SC2 committees. ROOT official support
In 1994, fundamental divergence of opinions in Application Software group in IT. The PAW/Geant3 team is dismantled.
Fortra
n9
0??
?
C++,Commercial Software
C++,
Open
Sourc
e
ROOT
Rene Brun Computing in HEP / INSTR02 11
Current Situation
Rapidly evolving from the Old Model to the New Model described later.
Painful transition to C++ in most major collaborations.
A few collaborations have experience with ODBMS systems (ie Objectivity)
But the days of Objectivity in HEP are counted.
Some limited experience with Geant4 Move to ROOT
Rene Brun Computing in HEP / INSTR02 12
Situation in the US
BaBar: pioneers with Objy. Complex situation.Kanga/Root used in small labs. Analysis with PAW. Growing Root community. G3-->G4
Glast: Use Gaudi + Root (G3 ->G4?) STAR: Root + MySQL + G3 Phenix: Root + Objy(catalog) +G3 Phobos: Root + Oracle + G3 JLAB: Complex situations with the small
expmts. Root in the main exp. G3. Attempts to use G4.
Rene Brun Computing in HEP / INSTR02 13
Situation in the US CDF: Root-based I/O and analysis (online +
offline). Final transition from Fortran to C++. File catalog in Oracle. G3, attempts to use G4
D0: home-grown I/O system (using CINT). Move from PAW to Root for data analysis. SAM (Oracle) catalog. G3
Minos: Root-based framework. G3 BTeV: ? Attempts to use G4 Miniboon: Fortran ?
Rene Brun Computing in HEP / INSTR02 14
Situation in Japan
Belle: Home grown tools in C++: - non-OO Data Management System Panther ADAMO-like structure with a C++ interface - Analysis Framework : "B.A.S.F." module and path structure dynamic link of modules and I/O drivers event-by-event parallel processing capability on
SMP - Communication tool over network : "NSM" shared memory/message passing capability over
network to be used for slow control
Linear Collider: C++/Root-based. G3
Rene Brun Computing in HEP / INSTR02 15
Situation at DESY
H1: Moving from Fortran-BOS to C++/Root in a BOS-compatible way. Use Oracle, G3.
ZEUS: Moving from Fortran to C++. Have used Objectivity. PAW analysis. Root growing. G3
Hera-B: Has moved from Fortran to C++/Root
Tesla: Moving from Fortran G3 to C++/G4
Rene Brun Computing in HEP / INSTR02 16
Situation with CERN non-LHC
Aleph, Opal, Delphi, L3: Fortran/Zebra/PAW/G3.
Compass: C++ home-grown system + Objectivity. PAW-->Root for analysis. G3
Harp: Objectivity, G4 AMS: C++ home-grown system. First
guinea pigs with Objectivity. Use PAW, Oracle and Root, G3
NA60: Root (Alice clone)
Rene Brun Computing in HEP / INSTR02 17
Situation with LHC
ALICE: Root framework + MySQL. G3 and G4. Alien. Abandoning G4. Interface to Fluka
ATLAS: Develop Athena framework (based on Gaudi). Objectivity & Root. G3. Experience with G4.
CMS: Orca/Cobra frameworks. Was based on Objectivity. Switching to Root. G3. Experience with G4.
LHCB: Gaudi framework with Root I/O. G3, starting with G4.
Rene Brun Computing in HEP / INSTR02 18
1 billion people surfing the
Web
How Much Data is Involved?
105
104
103
102
Level 1 Rate (Hz)
High Level-1 Trigger(1 MHz)
High No. ChannelsHigh Bandwidth(500 Gbit/s)
High Data Archive(5 PetaBytes/year)10 Gbits/s in Data base
LHCB
KLOE
HERA-B
CDF II
CDF
H1ZEUS
UA1
LEP
NA49ALICE
Event Size (bytes)
104 105 106
ATLASCMS
106
107
STAR
Rene Brun Computing in HEP / INSTR02 19
ALICE Event/100
Front View of a simulatedevent with only 1/100 of the
expected multiplicity
Rene Brun Computing in HEP / INSTR02 20
ALICE Event/100Side View of a simulatedevent with only 1/100 of the
expected multiplicity
Estimated size of one raw event = 40 Mbytes
Simulated event with hits = 1.5 Gbytes
Time to simulate one event = 24 hours
After L3, the DAQ will generate 1.25 GigaBytes/second
2 PetaBytes/year
Rene Brun Computing in HEP / INSTR02 21
LHC Computing - a Multi-Tier Model
Department
Desktop
CERN – Tier 0(CERN - Tier 1)
Tier 1 X Y
Z622 M
bps2.5 Gbps
622 M
bp
s
155
mbp
s155 mbps
Tier2 Lab a
Uni b Lab c
Uni n Organising Software:
"Grid-Middleware"
"Transparent" user access to applications and all data
'X''Y''Z': RAL, IN2P3, FNAL, BNL, FZK(?), . . .
Rene Brun Computing in HEP / INSTR02 22
People
Number of peopleand world-wide collaborations
have a big impacton the Computing
strategy
Rene Brun Computing in HEP / INSTR02 23
Worldwide Software Development
Code Management tools Is CVS sufficient ?
Concurrent development implications side-effects of unstable source code concepts of old/pro/new obsolete
Nightly builds Code inspection/Quality checks procedures
Bug reporting system HyperNews Web-based documentation & information
Rene Brun Computing in HEP / INSTR02 24
Move to OO programming
Data structuresin Fortran
common blocks
Bankssystems
Zebra,BosBanks-like
systems in C/C++Hand-written
I/O converters
True ObjectsCollections in C++
with built-inObject persistency and
automatic schema evolution
Typical scenario
Rene Brun Computing in HEP / INSTR02 25
Dynamic Linking
ApplicationExecutable Module
Experimentlibraries
Userlibraries
Generallibraries
A Shared Library can be linked dynamically to a running executable module
A Shared Library facilitates the development and maintenance phases
Rene Brun Computing in HEP / INSTR02 26
Whiteboard Data Communication
Class 1 Class 2
Class 3
Class 4
Class 5Class 6
Class 7
Class 8
Rene Brun Computing in HEP / INSTR02 27
Generic task handling
Det
Det
Det
Det
Det
Det
DetTaskTask
Task
TaskTask
Task
TaskTask
Task
TaskTask
Task
TaskTask
Task
TaskTask
Task
TaskTask
Task
Rene Brun Computing in HEP / INSTR02 28
event
From a simple nested parenthesis structure
((..)(..)(((..)(..)(…..))))To complex Objects graphs with internal and external references
In KBytes
BA
Rene Brun Computing in HEP / INSTR02 29
Lines of code
G4: 600,000 C++ lines
ROOT: 700,000 C++ lines
700 classes
30 sub-systems
ALICE: 600,000 C++ lines
+ROOT + G4 +..+..
Rene Brun Computing in HEP / INSTR02 30
A CORE system
Objects Dictionary: Introspection
GUI: Interpreter I/O subsystem
Graphics, Histograms Virtual file Catalog
High Level apps GRID toolkits
Experiment Software
Event Store Data bases
Rene Brun Computing in HEP / INSTR02 31
Data Bases: model-1
Put everything in an Object Data base like Objectivity or Oracle 9i
Choice of RD45 project Many experiments initially following this
line Abandonned by most experiments
recently Solution not suited for interactive analysis
Rene Brun Computing in HEP / INSTR02 32
Data Bases: model-2
Put write-once data in an object store like ROOT in Streamer mode
Use a RDBMS for : Run/Event catalogs Geometry, calibrations eg with ROOT<->Oracle interface
http://www.phenix.bnl.gov/WWW/publish/onuchin/rooObjy/
or with ROOT <-> Objectivity interface http://www.phenix.bnl.gov/WWW/publish/onuchin/RDBC/
Use ROOT split/no-split mode for data analysis
Combining2 technologies
ROOT
Oracle
Rene Brun Computing in HEP / INSTR02 33
The LHC Computing Grid Project
Next week at CERN
Rene Brun Computing in HEP / INSTR02 34
A ROOT-based CORE system
Objects Dictionary: Introspection
GUI: Interpreter I/O subsystem
Graphics, Histograms Virtual file Catalog
High Level apps GRID toolkits
Experiment Software
Event Store Data bases
agreed
Rene Brun Computing in HEP / INSTR02 35
ROOT + RDBMS Model
histograms
Calibrations
Geometries
Run/FileCatalog
Trees
Event Store
ROOTfiles
OracleMySQL
Rene Brun Computing in HEP / INSTR02 36
Memory <--> TreeThe Tree entry serial number
0123456789101112131415161718
T.Fill()
T.GetEntry(6)
T
Memory
Rene Brun Computing in HEP / INSTR02 37
Tree Friends
0123456789101112131415161718
0123456789101112131415161718
0123456789101112131415161718
Public
read
Public
read
User
Write
Entry # 8
Rene Brun Computing in HEP / INSTR02 38
Chains of Trees
0 0 0 0 0 00
11
0
4 52
45 4
7
0 12 17 23 26 32 37 45 49
TChain ch(“T”);
ch.Add(“f0.root”);
ch.Add(“f1.root”);
ch.Add(“f2.root”);
ch.Add(“f3.root”);
ch.Add(“f4.root”);
ch.Add(“f5.root”);
ch.Add(“f6.root”);
ch.Add(“f7.root”);
ch.GetEntry(28);
Binary search in table above
find slot 4, local entry 2
T.GetEntry(2) in f4.root
ch.GetEntryWithIndex(12,567);
0 7654321
f0 f2 f3 f4 f5 f6 f7f1
Rene Brun Computing in HEP / INSTR02 39
Automatic Schema Evolution
Rene Brun Computing in HEP / INSTR02 40
Auto Schema Evolution (2)
Rene Brun Computing in HEP / INSTR02 41
Self-describing files
Dictionary for persistent classes written to the file.
ROOT files can be read by foreign readers (JAS)
Support for Backward and Forward compatibility
Files created in 2002 must be readable in 2015
Classes (data objects) for all objects in a file can be regenerated via TFile::MakeProjectRoot >TFile f(“demo.root”);
Root > f.MakeProject(“dir”,”*”,”new++”);
Rene Brun Computing in HEP / INSTR02 42
Online/Offline
Groups traditionally separated. Growing synergy dictated by common
needs for data bases and event storage, GUI, graphics, analysis systems, L3 software, input from detector simulation programs, tools.
Inter-process objects and messages passing requires a common object dictionary.
Growing synergy imposed by the Data Challenges.
A good example: ALICE Data Challenges
Rene Brun Computing in HEP / INSTR02 43
Regional TIER 1 TIER 2
ALICE Data challenges
DAQ
ROOT I/OCERN TIER 0 TIER 1
Raw Data
Simulated Data GEANT3
GEANT4FLUKA
AliRoot
CASTOR
GRID
Performance Monitoring
ROOT
FileCatalogue
Rene Brun Computing in HEP / INSTR02 44
Writing to local diskMigration to tape
ALICE Data Challenge III Need to run yearly DC of increasing complexity and size to reach 1.25GB/s ADC III gave excellent system stability during 3 months
DATE throughput: 550 MB/s (max) 350 MB/s (ALICE-like) DATE+ROOT+CASTOR throughput: 120 MB/s, <85> MB/s 2200 runs, 2* 107 events, 86 hours, 54 TB DATE run 500 TB in DAQ, 200 TB in DAQ+ROOT I/O, 110 TB in CASTOR 105 files > 1GB in CASTOR and in MetaData DB
HP SMP’s: cost-effective alternative to inexpensive disk servers Online monitoring tools developed
0
500
1000
1500
2000
2500
3000
1999 2000 2001 2002 2003 2004 2005 2006
DAQMass Storage
0
50
100
150
200
250
300
350
1999 2000 2001
DAQ
Mass Storage (sust)
Mass Storage (peak)
MB
/s
MB
/s
Rene Brun Computing in HEP / INSTR02 45
CPU
Fantastic evolution, but not the only issue
Intel-compatible processors only ?
IA-64 processors ?
CERN units
Event-level parallelism
Rene Brun Computing in HEP / INSTR02 46
Disk
Disk bandwidth still a problem
IDE: 10->20 MB/s. Latency: > 5ms
GBytes
In 2007, cost of one Petabyte of disk or tape should be the same
Keep on disk DST/AOD as much as possible
Rene Brun Computing in HEP / INSTR02 47
GRID activities
Several Projects Starting in the US (iVDGL) and Europe (EDG)
No time to describe these projects.
Concentrate on one application
http://alien.cern.ch
Rene Brun Computing in HEP / INSTR02 48
ALICE GRID resources
Yerevan
CERN
Saclay
Lyon
Dubna
Capetown, ZA
Birmingham
Cagliari
NIKHEF
GSI
Catania
BolognaTorino
Padova
IRB
Kolkata, India
OSU/OSCLBL/NERSC
Merida
Bari
http://www.to.infn.it/activities/experiments/alice-grid
1000 physicists
37 people GRID-aware
21 institutions
Rene Brun Computing in HEP / INSTR02 49
ALICE GRID File Catalogue as a global file system on a RDB TAG Catalogue, as extension Secure Authentication
Interface to Globus available Central Queue Manager ("pull" vs "push" model)
Interface to EDG Resource Broker available Monitoring infrastructure
The CORE GRID functionality Automatic software installation with AliKit http://alien.cern.ch
Rene Brun Computing in HEP / INSTR02 50
ALIENalie n
SO AP Se rve r
AD M IN
P R O C ESSES
Autho r isatio nSe rvic e
allie n(she ll ,W e b)
C lie nt
D B IP ro xy se rve r
File C atalo gue
File C atalo gue
D BD rive r
File transpo r tSe rvic e
U se r Applic atio n(C /C + + /Java/P e r l)
SO AP C lie nt
D B SyncSe rvic e
D ISK
File catalogue : global file system
on top of relational database
Secure authentication service independent of underlying database
Central task queue
API
Services (file transport, sync)
Perl5
SOAP
Architecture
ALICEUSERS
ALICESIM
Tier1
ALICELOCAL
|--./| |--cern.ch/| | |--user/| | | |--a/| | | | |--admin/| | | | || | | | |--aliprod/| | | || | | |--f/| | | | |--fca/| | | || | | |--p/| | | | |--psaiz/| | | | | |--as/| | | | | || | | | | |--dos/| | | | | || | | | | |--local/
|--simulation/| |--2001-01/| | |--V3.05/| | | |--Config.C| | | |--grun.C
| |--36/| | |--stderr| | |--stdin| | |--stdout| || |--37/| | |--stderr| | |--stdin| | |--stdout| || |--38/| | |--stderr| | |--stdin| | |--stdout
| | | || | | |--b/| | | | |--barbera/
Files, commands (job specification) as well as job
input and output, tags and even binary package tar files are
stored in the catalogue
File catalogue
AD M IN
Autho r is atio nSe rvic e
allie n(she ll ,W e b)
C lie nt
D B IP ro xy se rve r
File C atalo gue
File C atalo gue
D BD rive r
Authentication
allie n(she ll ,W e b)
C lie nt
D B IP ro xy se rve r
File C atalo gue
File C atalo gue
D BD rive r
File transpo r tSe rvic e
D ISK
Data access
--./| |--r3418_01-01.ds| |--r3418_02-02.ds| |--r3418_03-03.ds| |--r3418_04-04.ds| |--r3418_05-05.ds| |--r3418_06-06.ds| |--r3418_07-07.ds| |--r3418_08-08.ds| |--r3418_09-09.ds| |--r3418_10-10.ds| |--r3418_11-11.ds| |--r3418_12-12.ds| |--r3418_13-13.ds| |--r3418_14-14.ds| |--r3418_15-15.ds
D0
pathdirhostIndexentryId
char(255)integer(11)integer(11)integer(11)
<fk><pk>
T2526
typedirnameownerctimecommentcontentmethodmethodArggownersize
char(4)integer(8)char(64)char(8)char(16)char(80)char(255)char(20)char(255)char(8)integer(11)
T2527
typedirnameownerctimecommentcontentmethodmethodArggownersize
char(4)integer(8)char(64)char(8)char(16)char(80)char(255)char(20)char(255)char(8)integer(11)
Bookkeeping
Rene Brun Computing in HEP / INSTR02 51
AliEn as a meta-GRID
AliEn User Interface
AliEn stackiVDGL stack EDG stack
Rene Brun Computing in HEP / INSTR02 52
DataGrid & PROOF
Local
Remote
Selection
Parameters
Procedure
Proc.C
Proc.C
Proc.C
Proc.C
Proc.C
PROOF
CPU
CPU
CPU
CPU
CPU
CPU
TagDB
RDB
DB1
DB4
DB5
DB6
DB3
DB2
Bring the KB to the PB and not the PB to the KB
Rene Brun Computing in HEP / INSTR02 53
Parallel Script Execution
root
Remote PROOF Cluster
proof
proof
proof
TNetFile
TFile
Local PC
$ root
ana.Cstdout/obj
node1
node2
node3
node4
$ root
root [0] .x ana.C
$ root
root [0] .x ana.C
root [1] gROOT->Proof(“remote”)
$ root
root [0] .x ana.C
root [1] gROOT->Proof(“remote”)
root [2] gProof->Exec(“.x ana.C”)
ana.C
proof
proof = slave server
proof
proof = master server
#proof.confslave node1slave node2slave node3slave node4
*.root
*.root
*.root
*.root
Rene Brun Computing in HEP / INSTR02 54
Conclusions: Not presented
Scripting languages (Python, CINT) GUI and graphics Detector Geometry Data Bases G3, G4, Fluka saga Conditions Data Bases Code Management tools on top of CVS Mathematical libraries Experiment Frameworks Progress expected
in these areasin the coming two years
Rene Brun Computing in HEP / INSTR02 55
Conclusions
Things converging rapidly in the past few months.
A common solution for I/O in view. Hybrid solution (Root-based event store +
RDBMS system(s) + GRID catalogs First GRID tools/systems appearing Discussions between major developers Still many problems to be solved.