GridPP Deployment Status Steve Traylen s.traylen@rl.ac.uk 28th October 2004 GOSC Face to Face, NESC, UK.
Post on 28-Mar-2015
216 Views
Preview:
Transcript
GridPP Deployment Status
Steve Traylen
Steve Traylens.traylen@rl.ac.uk
28th October 2004GOSC Face to Face, NESC, UK
Contents
• Middleware components of the GridPP Production System
• Status of the current operational Grid
• Future plans and challenges
• Summary
• GridPP 2 – From Prototype to Production
CMS LHCb ATLAS ALICE
1 Megabyte (1MB)A digital photo
1 Gigabyte (1GB) = 1000MBA DVD movie
1 Terabyte (1TB)= 1000GBWorld annual book production
1 Petabyte (1PB)= 1000TBAnnual production of one LHC experiment
1 Exabyte (1EB)= 1000 PBWorld annual information production
les.
rob
ert
son
@ce
rn.c
h
The physics driver
• 40 million collisions per second
• After filtering, 100-200 collisions of interest per second
• 1-10 Megabytes of data digitised for each collision = recording rate of 0.1-1 Gigabytes/sec
• 1010 collisions recorded each year = ~10 Petabytes/year of data
The LHC
The UK response
GridPPGridPP – A UK Computing Grid for
Particle Physics
19 UK Universities, CCLRC (RAL & Daresbury) and CERN
Funded by the Particle Physics and Astronomy Research Council (PPARC)
GridPP1 - Sept. 2001-2004 £17m "From Web to Grid“
GridPP2 – Sept. 2004-2007 £16(+1)m "From Prototype to Production"
Current context of GridPP
UK Core e-Science Programme
Institutes
Tier-2 Centres
CERNLCG
EGEE
GridPPTier-1/A
Middleware, Security,
Networking
Experiments
GridSupportCentre
Not to scale!
Apps Dev
AppsI nt
GridPP
UK Core e-Science Programme
Institutes
Tier-2 Centres
CERNLCG
CERNLCG
EGEE
GridPPGridPPTier-1/ATier-1/A
Middleware, Security,
Networking
Middleware, Security,
Networking
Experiments
GridSupportCentre
GridSupportCentre
Not to scale!
Apps DevApps Dev
AppsI nt
GridPP
Our grid is working …
NorthGrid ****Daresbury, Lancaster, Liverpool,Manchester, Sheffield
SouthGrid *Birmingham, Bristol, Cambridge,Oxford, RAL PPD, Warwick
ScotGrid *Durham, Edinburgh, Glasgow
LondonGrid ***Brunel, Imperial, QMUL, RHUL, UCL
… and is part of LCG
Resources are being used for data challenges
• Within the UK we have some VO/experiment Memorandum of Understandings in place
• Tier-2 structure is working well
Scale
GridPP prototype Grid> 1,000 CPUs
– 500 CPUs at the Tier-1 at RAL
> 500 CPUs at 11 sites across UK organised in 4 Regional Tier-2s
> 500 TB of storage> 800 simultaneous jobs
• Integrated with international LHC Computing Grid (LCG)
> 5,000 CPUs> 4,000 TB of storage> 85 sites around the world> 4,000 simultaneous jobs• monitored via Grid Operations
Centre (RAL)
CPUs FreeCPUs
RunJobs
WaitJobs
Avail TB Used TB Max CPU
Ave.CPU
Total 8049 1439 6610 8733 3558.47 1273.86 9148 6198
http://goc.grid.sinica.edu.tw/gstat/
(hyperthreading enabled on some sites)
Operational status (October)
Site Version Sept Version OcttotalCPU
SepttotalCPU
OctseAvail TB Sept
seAvail Oct
maxCPU Sept
MaxCPU Oct
avgCPU Sept
avgCPU Oct
1 CAVENDISH-LCG2 LCG-2_1_1 LCG-2_1_1 11 11 1.59 0.01 11 11 10 102 IC-LCG2 LCG-2_2_0 LCG-2_2_0 66 66 0.4 0.4 66 66 63 653 RAL-LCG2 LCG-2_2_0 LCG-2_2_0 494 482 934.13 937.76 494 494 482 4694 BHAM-LCG2 LCG-2_2_0 LCG-2_2_0 20 22 0.03 0.03 20 22 18 205 BitLab-LCG2 LCG-2_2_0 na 1 0 0.09 0 1 1 0 06 Lancs-LCG2 LCG-2_1_1 LCG-2_1_1 24 26 1.85 1.85 27 26 23 227 LivHEP-LCG2 LCG-2_1_1 LCG-2_1_1 94 113 0.09 0.09 94 113 51 968 ManHEP-LCG2 LCG-2_1_1 LCG-2_2_0 98 98 0.09 0.09 98 98 97 939 OXFORD-01-LCG2 LCG-2_1_1 LCG-2_1_1 72 32 1.49 1.48 72 72 69 39
10 QMUL-eScience LCG-2_1_0 LCG-2_1_0 576 566 0.09 0.1 576 576 563 56511 RALPP-LCG LCG-2_2_0 LCG-2_2_0 88 88 0.13 0.26 88 88 52 8212 RHUL-LCG2 LCG-2_1_1 LCG-2_1_1 144 144 5.43 8.16 148 148 90 11413 ScotGRID-Edinburgh LCG-2_0_0 LCG-2_0_0 1 1 4.76 0.87 1 1 0 014 scotgrid-gla LCG-2_2_0 LCG-2_2_0 2 3 0 0.09 2 3 1 215 SHEFFIELD-LCG2 LCG-2_1_1 LCG-2_1_1 39 0 0.42 0.42 43 39 36 2216 UCL-CCC LCG-2_2_0 LCG-2_2_0 340 356 0.49 0.49 356 356 299 35217 UCL-HEP LCG-2_1_1 LCG-2_1_1 76 76 1.85 1.85 76 76 64 54
VOs active
Site alice atlas babar cms dteam dzero hone lhcb sixt zeus TOTAL 101 CAVENDISH-LCG2 1 1 1 1 1 52 IC-LCG2 1 1 1 1 1 1 63 RAL-LCG2 1 1 1 1 1 1 1 1 1 94 BHAM-LCG2 1 1 1 1 1 1 65 BitLab-LCG2 1 1 1 1 1 56 Lancs-LCG2 1 1 1 1 1 57 LivHEP-LCG2 1 1 1 1 48 ManHEP-LCG2 1 1 1 1 1 1 1 79 OXFORD-01-LCG2 1 1 1 1 1 5
10 QMUL-eScience 1 1 1 1 1 1 1 711 RALPP-LCG 1 1 1 1 1 1 1 1 1 912 RHUL-LCG2 1 1 1 1 1 1 613 ScotGRID-Edinburgh 1 1 1 314 scotgrid-gla 1 1 1 1 415 SHEFFIELD-LCG2 1 1 1 1 1 1 616 UCL-CCC 1 1 1 1 1 1 617 UCL-HEP 1 1 2
TOTAL 13 17 5 13 17 3 2 16 6 3
Who is directly involved?
Number Position Status
General
1 Production manager In place and engaged
1 Applications expert Identified but not formally engaged
2 Tier-1 /deployment expert In place and fully engaged
4 Tier-2 coordinators In place and functioning well
0.5 VO management Will be part time but not yet in place
9.0 Hardware support Post allocated but not yet filled
Specialist
1 Data and storage management Existing expert
1 Work load management Existing expert
1 Security officer Not yet recruited
1 Networking Starting in September
Past upgrade experience at RAL
CSF Linux CPU Use 2001-02
0
20000
40000
60000
80000
100000
120000
140000
Jan
-0
1
Feb
-0
1
Mar-
01
Ap
r-0
1
May-
01
Jun
-0
1
Jul-
01
Sep
-0
1
Oct-
01
No
v-
01
Dec-
01
Jan
-0
2
Feb
-0
2
Ap
r-0
2
May-
02
Jun
-0
2
Jul-
02
Au
g-
02
cpu
Previously utilisation of new resources grew steadily over weeks or months.
Tier-1 update 27-28th July 2004
Hardware Upgrade
With the Grid we see a much more rapid utilisation of newly deployed resources.
The infrastructure developed in EDG/GridPP1
Job submissionPython – default
Java – GUIAPIs (C++,J,P)
Batch workers
StorageElement
Gatekeeper (PBS Scheduler)
GridFTP Server
NFS, Tape, Castor
User Interface
ComputingElement
Resource broker(C++ Condor MM libraries, Condor-G for submission)
Replica catalogue per VO (or equiv.)
Berkely DatabaseInformation Index
AA server(VOMS)
UIJDL
Logging & Book keepingMySQL DB – stores job state info
Common Grid Components
• LCG uses middleware common to other Grid Projects.– VDT (v1.1.14)
• Globus Gatekeeper.• Globus MDS.• GlueCE Information Provider.• Used by NGS, Grid3 and Nordugrid.
• Preserving this core increases chances of inter grid interoperability.
Extra Grid Components
• LCG extends VDT with fixes and the deployment of other grid services.
• This is only done when there is a shortfall or performance issue with the existing middleware.
• Most are grid wide services for LCG rather than extra components for sites to install.– Minimise conflicts between grids.– Not always true – see later.
LCG PBSJobManager
• Motivation– Standard Globus JobManager starts one
perl process per job, queued or running.• One user can destroy a Gatekeeper easily.
– Also assumes a shared /home file system is present.• Not scalable to 1000s of nodes.• NFS a single failure point.
– The Resource Broker must poll jobs indivdually.
LCG PBSJobManager
• Solution– LCG jobmanager stages files to batch
worker with scp and GridFTP.
• Creates new problems though.– Even harder to debug and there is more
to go wrong.– MPI jobs more difficult though an rsync
work around exists.
LCG PBSJobManager
• Solution– JobManager starts up a “GridMonitor” on
the gatekeeper. – One GridMonitor per Resource Broker is
started currently.– Resource Broker communicates with the
monitor instead of polling jobs individually.– Moving this to one GridMonitor per user is
possible.
• Currently deployed at almost all GridPP sites.
Storage in LCG
• Currently there are three active solutions.– GridFTP servers, the so called ClassicSE– SRM interfaces at CERN, IHEP(Russia), DESY
and RAL (this week).– edg-se – Only one as a front end the atlas
data store tape system at RAL.
• The edg-rm and lcg-* commands abstract the end user from these interfaces.
Storage - SRM
• SRM = Storage Resource Manager.• Motivation
– Sites need to move files around and reorganise data dynamically.
– The end user wants/requires a consistent name space for their files.
– End users want to be able to reserve space this space as well.
• SRM will in time be the preferred solution supported within LCG.
SRM Deployment
• Current storage solution for LCG is dCache with an SRM interface. Produced by DESY and FNAL.
• This is currently deployed at RAL in a test state and is slipping into production initially for the CMS experiment.
• Expectation is that dCache with SRM will provide a solution for many sites.– Edinburgh, Manchester, Oxford all keen to
deploy.
SRM/dCache at RAL
Resource Broker
• Allows selection of and submission to sites based on what they publish into the information system.
• Queues are published with– Queue lengths– Software available.– Authorised VOs or individual DNs.
• The RB can query the replica catalogue to run at a site with a particular file.
• Three RBs are deployed in the UK.
L&B
• L&B = Logging and Bookkeeping Service
• Jobs publish their Grid State to L&B.– Either by calling commands installed on
batch worker.– Or by GridFTP’ing the job wrapper back.
• The second requires no software on batch workers but the first gives better feedback.
Application Installation with
LCG• Currently a sub VO of software
managers owns an NFS mounted space.– Software area managed by jobs.– Software validated in process.– The drop a status file on to the file which is
published by the site.• With the RB
– End users match jobs to tagged sites.– SW managers install SW at non tagged
sites.• This is being extended to allow DTEAM
to install grid clients SW on WNs.
R-GMA
• Developed by GridPP within both EDG and now EGEE.
• Takes the role of a grid enabled SQL database.
• Example applications include CMS and D to publish their job bookkeeping.
• Can also be used to transport the Glue values and allows SQL lookups of Glue.
• R-GMA is deployed at most UK HEP sites.• RAL currently runs the single instance of the
R-GMA registry.
Next LCG Release
• LCG 2_3_0 is due now.– Built entirely on SL3 (RHE3 clone).
• RH73 still an option.
– Many stability improvements.– Addition of accounting solution.– Easier addition of VOs.– Addition of DCache/SRM.
• and lots more…
• This release will last into next year.• Potentially the last release before gLite
components appear.
There are still challenges
• Middleware validation
• Meeting experiment requirements with the Grid
• Distributed file (and sub-file) management
• Experiment software distribution
• Production accounting
• Encouraging an open sharing of resources
• Security
• Smoothing deployment and service upgrades.
Middleware validation
CERTIFICATIONTESTING
Integrate
BasicFunctionality
Tests
Run testsC&T suitesSite suites
RunCertification
Matrix
Releasecandidate
tag
APPINTEGR
Certifiedrelease
tag
DE
VE
LO
PM
EN
T &
INT
EG
RA
TIO
NU
NIT
& F
UN
CT
ION
AL
TE
ST
ING
DevTag
JRA1
HEPEXPTS
BIO-MED
OTHERTBD
APPSSW
Installation
DE
PL
OY
ME
NT
PR
EP
AR
AT
ION
Deploymentrelease
tag
DEPLOY
SA1
SERVICES
PR
E-P
RO
DU
CT
ION
PR
OD
UC
TIO
N
Productiontag
Is starting to be addressed through a Certification and Testing testbed…RAL is involved with both JRA1 and Pre Production systems.
• ATLAS Data Challenge to validate world-wide computing model
• Packaging, distribution and installation: Scale:one release build takes 10 hours produces 2.5 GB of files
• Complexity: 500 packages, Mloc, 100s of developers and 1000s of users– ATLAS collaboration
is widely distributed:140 institutes, all wanting to use the software
– needs ‘push-button’ easy installation..
Physics Models
Monte Carlo Truth DataMonte Carlo Truth Data
MC Raw DataMC Raw Data
Reconstruction
MC Event Summary DataMC Event Summary Data MC Event Tags MC Event Tags
Detector Simulation
Raw DataRaw Data
Reconstruction
Data Acquisition
Level 3 trigger
Trigger TagsTrigger Tags
Event Summary Data
ESD
Event Summary Data
ESD Event Tags Event Tags
Calibration DataCalibration Data
Run ConditionsRun Conditions
Trigger System
Step 1: Monte Carlo
Data Challenges
Step 1: Monte Carlo
Data Challenges
Step 2: Real DataStep 2: Real Data
Software distribution
Summary
• The Large Hadron Collider data volumes make Grid computing a necessity
• GridPP1 with EDG developed a successful Grid prototype
• GridPP members have played a critical role in most areas – security, work load management, information systems, monitoring & operations.
• GridPP involvement continues with the Enabling Grids for e-SciencE (EGEE) project – driving the federating of Grids
• As we move towards a full production service we face many challenges in areas such as deployment, accounting and true open sharing of resources
Useful links
GRIDPP and LCG:
• GridPP collaborationhttp://www.gridpp.ac.uk/
• Grid Operations Centre (inc. maps)http://goc.grid-support.ac.uk/
• The LHC Computing Gridhttp://lcg.web.cern.ch/LCG/
Others
• PPARChttp://www.pparc.ac.uk/Rs/Fs/Es/intro.asp
• The EGEE projecthttp://cern.ch/egee/
• The European Data Grid final reviewhttp://eu-datagrid.web.cern.ch/eu-datagrid/
top related