10 January 2005 Oversight Committee Tony Doyle - University of Glasgow GridPP: status report Andy Parker (Taken from GridPP report at OSC by Tony Doyle)
Jan 19, 2016
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
GridPP: status report
Andy Parker
(Taken from GridPP report at OSC by Tony Doyle)
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
What was GridPP1?
• A team that built a working prototype grid of significant scale
> 2,000 (9,000) CPUs> 1,000 (5,000) TB of available storage> 1,000 (6,000) simultaneous jobs
• A complex project where 88% of the milestones were completed and all metrics were within specification
H I H I H H H H H
J 1. 1 K J 2. 1 K J 3. 1 K J 4. 1 K J 5. 1 K J 6. 1 K J 7. 1 K1.1.1 1.1.2 1.1.3 1.1.4 2.1.1 2.1.2 2.1.3 2.1.4 3.1.1 3.1.2 3.1.3 3.1.4 4.1.1 4.1.2 4.1.3 4.1.4 5.1.1 5.1.2 5.1.3 6.1.1 6.1.2 6.1.3 6.1.4 7.1.1 7.1.2 7.1.3 7.1.41.1.5 2.1.5 2.1.6 2.1.7 2.1.8 3.1.5 3.1.6 3.1.7 3.1.8 4.1.5 4.1.6 4.1.7 4.1.8 6.1.5
2.1.9 3.1.9 3.1.10 4.1.9
J 1. 2 K J 2. 2 K J 3. 2 K J 4. 2 K J 5. 2 K J 6. 2 K J 7. 2 K1.2.1 1.2.2 1.2.3 1.2.4 2.2.1 2.2.2 2.2.3 2.2.4 3.2.1 3.2.2 3.2.3 3.2.4 4.2.1 4.2.2 4.2.3 4.2.4 5.2.1 5.2.2 5.2.3 6.2.1 6.2.2 6.2.3 7.2.1 7.2.2 7.2.31.2.5 1.2.6 1.2.7 1.2.8 2.2.5 2.2.6 2.2.7 3.2.5 3.2.6 3.2.7 3.2.8 4.2.5 4.2.6 4.2.71.2.9 1.2.10 3.2.9
J 1. 3 K J 2. 3 K J 3. 3 K J 4. 3 K J 5. 3 K J 6. 3 K J 7. 3 K1.3.1 1.3.2 1.3.3 1.3.4 2.3.1 2.3.2 2.3.3 2.3.4 3.3.1 3.3.2 3.3.3 3.3.4 4.3.1 4.3.2 4.3.3 4.3.4 5.3.1 5.3.2 5.3.3 6.3.1 6.3.2 6.3.3 6.3.4 7.3.1 7.3.2 7.3.3 7.3.41.3.5 1.3.6 1.3.7 1.3.8 2.3.5 2.3.6 2.3.7 3.3.5 3.3.6 4.3.51.3.9 1.3.101.3.11
J 1. 4 K J 2. 4 K J 3. 4 K J 4. 4 K J 5. 4 K1.4.1 1.4.2 1.4.3 1.4.4 2.4.1 2.4.2 2.4.3 2.4.4 3.4.1 3.4.2 3.4.3 3.4.4 4.4.1 4.4.2 4.4.3 4.4.4 5.4.1 5.4.2 5.4.3 5.4.41.4.5 1.4.6 1.4.7 1.4.8 2.4.5 2.4.6 2.4.7 3.4.5 3.4.6 3.4.7 3.4.8 4.4.5 4.4.6 5.4.51.4.9 3.4.9 3.4.10 Metric OK 1.1.1
Metric not OK 1.1.1J 1. 5 K J 2. 5 K J 3. 5 K J 4. 5 K Task complete 1.1.1
1.5.1 1.5.2 1.5.3 1.5.4 2.5.1 2.5.2 2.5.3 2.5.4 3.5.1 3.5.2 3.5.3 3.5.4 4.5.1 4.5.2 4.5.3 4.5.4 Task overdue 1.1.11.5.5 1.5.6 1.5.7 1.5.8 2.5.5 2.5.6 2.5.7 3.5.5 3.5.6 3.5.7 60 days 1.1.11.5.9 1.5.10 Task not due soon 1.1.1
Not Active 1.1.1J 2. 6 K J 3. 6 K J 4. 6 K No Task or metric
2.6.1 2.6.2 2.6.3 2.6.4 3.6.1 3.6.2 3.6.3 3.6.4 4.6.1 4.6.2 4.6.32.6.5 2.6.6 2.6.7 2.6.8 3.6.5 3.6.6 3.6.7 3.6.8 Navigate up J2.6.9 3.6.9 3.6.103.6.11 3.6.12 Navigate down K
External link IJ 2. 7 K J 3. 7 K Link to goals H
2.7.1 2.7.2 2.7.3 2.7.4 3.7.1 3.7.2 3.7.3 3.7.42.7.5 2.7.6 2.7.7 2.7.8 3.7.5 3.7.6
J 2. 8 K J 3. 8 K2.8.1 2.8.2 2.8.3 2.8.4 3.8.1 3.8.2 3.8.32.8.5
WP8
1 2 3
Deployment
WP4
WP5
Fabric
Technology
WP6
Due within
ATLAS
GridPP Goal
ResourcesInteroperability Dissemination
Tier-1
Tier-A
LHCb Tier-2
CERN DataGrid Applications Infrastructure
WP1
WP2
WP3
LCG Creation
Applications
WP7
ATLAS/LHCb
CMS
BaBar
CDF/DO
UKQCD
Other
Data Challenges
Rollout
Testbed
1-Jan-04Status Date
Int. Standards
Open Source
Worldwide Integration
UK Integration
Monitoring
DevelopingEngagement
Participation
To develop and deploy a large scale science Gridin the UK for the use of the Particle Physics community
Presentation Deployment
5 6 74
Update
Clear
A Success
“The achievement of
something desired, planned, or
attempted”
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
What is GridPP2?
X 0. Production Grid X
X X X X X X
X 1. 1 X X 2.1 X X 3.1 X X 4.1 X X 5.1 X X 6.1 X
X 1. 2 X X 2.2 X X 3.2 X X 4.2 X X 5.2 X X 6.2 X
X 1. 3 X X 2.3 X X 3.3 X X 4.3 X X 6.3 X
X 1. 4 X X 2.4 X X 3.4 X X 4.4 X X 6.4 X
X 2.5 X X 3.5 X X 4.5 X
Navigate down XExternal link XLink to goals X
X 2.6 X X 3.6 X X 4.6 X
Network
Management
& MonitoringInformation PhenoGrid
KnowledgeTransfer
32
Management
Grid Deployment Security CMS UKQCD
Engagement
Grid Technology Workload LHCb D0
Computing Fabric Data & Storage Ganga CDF Deployment
Grid Operations
1 6M/S/N LHC Apps
54
GridPP2 GoalTo develop and deploy a large scale production quality grid in the UK for the use of the Particle Physics community
Tier-A Tier-1 Tier-2 Deployment Middleware Support Experiment Support
Interoperability
ATLAS Dissemination
Management ExternalLCG
Planning
Applications Metadata
Non-LHC Apps
BaBar
SAMGrid
LHC Deployment Portal
Structures agreed and in place (except LCG phase-2)
•253 Milestones, 112 Monitoring Metrics at present.•Must deliver a “Production Grid”: robust, reliable, resilient, secure, stable service delivered to end-user applications. •The Collaboration aims to develop, deploy and operate a very large Production Grid in the UK for use by the worldwide particle physics community.
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
LCG Overview
By 2007:- 100,000 CPUs- More than 100 institutes worldwide
- building on complex middleware being developed in advanced Grid technology projects, both in Europe (Glite) and in the USA (VDT)
- prototype went live in September 2003 in 12 countries
- Extensively tested by the LHC experiments during this summer
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
LHCb Data Challenge
424 CPU years (4,000 kSI2k months), 186M events • UK’s input significant (>1/4 total) • LCG(UK) resource:
– Tier-1 7.7%– Tier-2 sites:– London 3.9%– South 2.3%– North 1.4%
• DIRAC:– Imperial 2.0%– L'pool 3.1%– Oxford 0.1%– ScotGrid 5.1%
DIRAC alone
LCG inaction
1.8 106/day
LCG paused
Phase 1 Completed
3-5 106/dayLCG
restarted
186 M Produced Events
Entering Grid Production Phase..
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
Paradigm ShiftTransition to Grid…
Jun: 80%:20%
25% of DC’04
Aug: 27%:73%
42% of DC’04
May: 89%:11%
11% of DC’04
Jul: 77%:23%
22% of DC’04
424 CPU · Years
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
Issues
https://edms.cern.ch/file/495809/2.2/LCG2-Limitations_and_Requirements.pdf
First large-scale Grid production problems being addressed…at all levels
“LCG-2 MIDDLEWARE
PROBLEMS ANDREQUIREMENTS
FOR LHC EXPERIMENT DATA
CHALLENGES”
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
UK Tier-1/A Centre Rutherford Appleton
Laboratory
• High quality data services• National and international
role• UK focus for international
Grid development1000 CPU200 TB Disk60 TB Tape
(Capacity 1PB)
Grid Resource Discovery Time = 8 Hours
2004 CPU Utilisation2004 Disk Use
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
UK Tier-2 Centres
ScotGridDurham, Edinburgh, Glasgow NorthGridDaresbury, Lancaster, Liverpool,Manchester, Sheffield
SouthGridBirmingham, Bristol, Cambridge,Oxford, RAL PPD, Warwick
LondonGridBrunel, Imperial, QMUL, RHUL, UCL
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
Level-2 Grid
In future will include services to facilitate collaborative (grid) computing•Authentication (PKI X509)•Job submission/batch service•Resource brokering•Authorisation•Virtual Organisation management •Certificate management•Information service•Data access/integration (SRB/OGSA-DAI/DQPS)•National Registry (of registry’s)•Data replication•Data caching•Grid monitoring•Accounting
* LeedsManchester *
* OxfordRAL *
* DL
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
Application Development
ATLAS LHCb CMS
BaBar (SLAC) SAMGrid (FermiLab)QCDGrid PhenoGrid
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
More Applications
ZEUS uses LCG•needs the Grid to respond to increasing demand for MC production• 5 million Geant events on Grid since August 2004
QCDGrid• For UKQCD • Currently a 4-site data grid • Key technologies used
- Globus Toolkit 2.4- European DataGrid- eXist XML database
•managing a few hundred gigabytes of data
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
Disseminationmuch has happened..
more people are reading about it..
LHCb-UK members get up to speed with the Grid Wed 5 Jan 2005GridPP in Pittsburgh Thu 9 Dec 2004GridPP website busier than ever Mon 6 Dec 2004Optorsim 2.0 released Wed 24 Nov 2004ZEUS produces 5 million Grid events Mon 15 Nov 2004CERN 50th anniversary reception Tue 26 Oct 2004GridPP at CHEP'04 Mon 18 Oct 2004LHCb data challenge first phase a success for LCG and UK Mon 4 Oct 2004Networking in Nottingham - GLIF launch meeting Mon 4 Oct 2004GridPP going for Gold - website award at AHM Mon 6 Sep 2004GridPP at the All Hands Meeting Wed 1 Sep 2004R-GMA included in latest LCG release Wed 18 Aug 2004LCG2 administrators learn tips and tricks in Oxford Tue 27 Jul 2004Take me to your (project) leader Fri 2 Jul 2004ScotGrid's 2nd birthday: ScotGrid clocks up 1 million CPU hours Fri 25 Jun 2004Meet your production manager Fri 18 Jun 2004GridPP10 report and photographs Wed 9 Jun 2004CERN recognizes UK's outstanding contribution to Grid computing Wed 2 Jun 2004UK particle physics Grid takes shape Wed 19 May 2004A new monitoring map for GridPP Mon 10 May 2004Press reaction to EGEE launch Tue 4 May 2004GridPP at the EGEE launch conference Tue 27 Apr 2004LCG2 released Thu 8 Apr 2004University of Warwick joins GridPP Thu 8 Apr 2004Grid computing steps up a gear: the start of EGEE Thu 1 Apr 2004EDG gets glowing final review Mon 22 Mar 2004Grids and Web Services meeting, 23 April, London Tue 16 Mar 2004EU DataGrid Software License approved by OSI Fri 27 Feb 2004GridPP Middleware workshop, March 4-5 2004, UCL Fri 20 Feb 2004Version 1.0 of the Optorsim grid simulation tool released by EU DataGrid Tue 17 Feb 2004Summary and photographs of the 9th GridPP Collaboration Meetin Thu 12 Feb 2004
138,976 hitsin December
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
Grid and e-Science Support
in 2008What areas require support?IV Running the Tier-1 Data CentreIV Hardware annual upgrade
IV Contribution to Tier-2 Sysman effort (non-PPARC) hardware IV Frontend Tier-2 hardware
IV Contribution to Tier-0 support
III One M/S/N expert in each of 6 areasIII Production manager and four Tier-2 coordinators
II Application/Grid experts (UK support)
I ATLAS Computing MoU commitments and supportI CMS Computing MoU commitments and supportI LHCb Core Tasks and Computing SupportI ALICE Computing supportI Future experiments adopt e-Infrastructure methods
• No GridPP management: (assume production mode established + devolved management to Institutes)
III. Grid Middleware
I. Experiment Layer
II. Application Middleware
IV. Facilities and Fabrics
10 January 2005 Oversight Committee Tony Doyle - University of Glasgow
OSC actions• Plan for support of physics analysis
community in 2007• Better understanding of capacity limits
at Tier1 and strategic priorities• £1M bid for LCG2 to go to PPARC
Science Committee• Plan to raise job completion efficiency• Statement on gLite• OSC views project as excellently
managed and very successful