1D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Scientific Data Management:An Incomplete
Experimental HENP Perspective
D. Olson, LBNL
26 March 2002SDM-ISIC Meeting
Gatlinburg
2D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Particle Physics Data Grid
www.ppdg.net
PI’s:Mount,Livny,Newman
Coordinators:Pordes,
Olson
3D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Contents
• Quick overview of HENP data—Generic data flow—Sizes, timescales—Average physicist view
• What’s hard—Making technology work in production—A clear view for average physicist—Analysis of large datasets—Other things as well
• Today, many issues wrapped in hopes for “Data Grid”
4D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Experimental HENP event data
• Basic character of data is “event”—May be few particles
5D. Olson, SDM-ISIC Mtg, 26 Mar 2002
BaBar event
http://www.slac.stanford.edu/BFROOT/
6D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Experimental HENP event data
• Basic character of data is “event”—May be few particles—May be MANY particles
7D. Olson, SDM-ISIC Mtg, 26 Mar 2002
STAR event, Au + Au
www.star.bnl.gov
8D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Experimental HENP event data
• Basic character of data is “event”—May be few tracks—May be MANY tracks
• Detector characteristics, beam types, triggers effect the type of events recorded
• Physics analysis is a statistical analysis of many (1000’s, M’s, B’s, T’s) independent events
9D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Generic data flow in HENP
“Skims”, “microDST production”, …
Filtering chosen to make this a convenient size
10D. Olson, SDM-ISIC Mtg, 26 Mar 2002
A collaboration of people
$100M, 10 yr, 100 people
Free?, 10 yr, 20 people
Free?, 1 yr, 10 people, 5x/yr
Free?, 1 mo, 1 person, 50x/yr
(“Typical” example today, LHC is larger)
11D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Example: CMS Tiers
12D. Olson, SDM-ISIC Mtg, 26 Mar 2002
List of major accelerator-based HENP experiments
Experiment Location # physicists Time scale
BaBar SLAC 800 1999 - 2010
STAR BNL / RHIC 450 2000 - 2010
PHENIX BNL / RHIC 450 2000 - 2010
Jlab/CLAS JLAB 200 2000 - 2010
CDF FNAL 800 1995 - 2010
D0 FNAL 800 1995 - 2010
ATLAS CERN 2000 2006 - 2016
CMS CERN 2000 2006 - 2016
ALICE CERN 1200 2007 - 2017
Jlab Hall D JLAB 200 2008 - 2018
13D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Size / frequency of basic activities
Item Size (TB) / Frequency (/yr)
Typical today LHC era (>5 yr)
Raw data 100 TB / yr 1,000 TB / yr
Event Reconstruction
3 / yr 2 / yr
DST data 1 > DST/ raw > 0.1 0.1 > DST/ raw > 0.02
microDST production
0.1 > microDST/DST > .001
?
Physics analysis 10 - 100 * #physicists / year
?
14D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Average physicist view
• Mythology, culture, terminology varies a lot from one experiment to another.
• BaBar— Object view or primary event store (Objectivity)— Event collection objects give primary access points to data
• Event collection has list of references to all event components of interest
• With 100,000 collections, how to organize them?— Ntuples & PAW for final data format, analysis tool
• STAR (first year data, getting started)
— A “production, trigger” is all reconstructed events for a trigger type with a certain version of code, (P00hg, central)
— Access point is list of directory path’s below which all data are stored on disk
— WZ will be setting up STACS— ROOT for data format and analysis tool
• …
15D. Olson, SDM-ISIC Mtg, 26 Mar 2002
What’s hard I, living with technology
• Typical computer center today— A couple STK Powderhorn tape silos, HPSS or home-grown
MSS— 1000 linux processors— Assortment of 100/1000 Gbps network— 50 TB disk (1000 spindles)— Network s/w for I/O (NFS, Objy AMS, RFIO, …)— AFS for distributed collaboration
• Can make large RAID filesystems w/ network access— Faults can affect many nodes
• stale NFS file handles• AFS faults affects nodes across country, work
— Large RAID is $$$• Desire to reduce effect of faults
— Fewer faults— More tolerance
• …
16D. Olson, SDM-ISIC Mtg, 26 Mar 2002
What’s hard II,A clear view for average
physicist
What’s going on in this box?
17D. Olson, SDM-ISIC Mtg, 26 Mar 2002
What’s hard II,A clear view for average
physicist
• What data is available?—“data” means
• List of files? (like STAR)• Collection object w/ pointers to all events? (like BaBar)
—“available” means• On disk? Where?• Exists?
• Does it really have the filters and calibrations I need?
• Is it the “official” version of the data?• …
18D. Olson, SDM-ISIC Mtg, 26 Mar 2002
What’s hard III,Analysis of large datasets
Dataset does not fit on disk, or requires parallel processing, or
is large enough operation that chance of fault is high
19D. Olson, SDM-ISIC Mtg, 26 Mar 2002
What’s hard III,Analysis of large datasets
• Dataset does not fit on disk—Needs access s/w to couple w/ processing
• SAM, STACS
—Does performance meet demand?
20D. Olson, SDM-ISIC Mtg, 26 Mar 2002
SAM (Sequential data Access via Meta-data)
http://d0db.fnal.gov/sam/
21D. Olson, SDM-ISIC Mtg, 26 Mar 2002
STACS
http://sdm.lbl.gov/projectindividual.php?ProjectID=STACS
22D. Olson, SDM-ISIC Mtg, 26 Mar 2002
What’s hard III,Analysis of large datasets
• Dataset does not fit on disk—Needs access s/w to couple w/ processing
• SAM, STACS
—Does performance meet demand?
• Needs parallel processing (not very hard)—Can not do analysis on private/personal machine—Schedule access to shared resource (CPU and disk)
• Operation for a single analysis is large enough that faults occur—Need exception handling—Need workflow management to complete failed
tasks or, at least, accurately report status
23D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Example shared nothing cluster
http://www.ihep.ac.cn/~chep01/paper/4-026.pdf
24D. Olson, SDM-ISIC Mtg, 26 Mar 2002
PPDG
25D. Olson, SDM-ISIC Mtg, 26 Mar 2002
Summary
• Faulty technology sets boundary conditions—Fault tolerant will expand boundaries of capabilities
• Data management is coupled with processing—Visualization (access w/o processing) is minor in
HENP—Need access to data when & where it is needed for
processing
• Working on data grid as context for data management
• PPDG has SDM ISIC as one of the technology base projects