U.S. ATLAS Project Overview
John HuthJohn Huth
Harvard UniversityHarvard University
LHC Computing ReviewLHC Computing Review
FNALFNALNovember 2001November 2001
November 01November 01John Huth John Huth , LHC Computing 2
Outline
International and U.S. ATLASInternational and U.S. ATLAS Organization U.S. ATLAS External Groups
Project Management PlanProject Management Plan
MilestonesMilestones
StatusStatus Software Facilities Grid efforts Physics
Funding ProfileFunding Profile Impact
November 01November 01John Huth John Huth , LHC Computing 3
International ATLAS
Computing Oversight BoardComputing Oversight Board
Computing Steering GroupComputing Steering Group Matrix of detector/task orientation
PBS structure gives tasks, schedules, resource loadingPBS structure gives tasks, schedules, resource loading Maps directly onto U.S. ATLAS WBS
Planning officer is now Torre Wenaus (U.S. ATLAS SW Mgr.)
Software deliverables delineated in Software AgreementsSoftware deliverables delineated in Software Agreements
Major milestones associated with releases, data challengesMajor milestones associated with releases, data challenges
Data Challenge coordinator: Gilbert Poulard (CERN)Data Challenge coordinator: Gilbert Poulard (CERN)
November 01November 01John Huth John Huth , LHC Computing 4
ATLAS Detector/Task matrix
Offline Offline
CoordinatorCoordinator
ReconstructionReconstruction SimulationSimulation DatabaseDatabase
ChairChair N. McCubbinN. McCubbin D. RousseauD. Rousseau K. AmakoK. Amako D. MalonD. Malon
Inner DetectorInner Detector D. BarberisD. Barberis D. RousseauD. Rousseau F. LuehringF. Luehring S. BentvelsenS. Bentvelsen
Liquid ArgonLiquid Argon J. CollotJ. Collot S. RajagopalanS. Rajagopalan M. LeltchoukM. Leltchouk S. SimionS. Simion//
R. SobieR. Sobie
Tile CalorimeterTile Calorimeter A. SolodkovA. Solodkov F. MerrittF. Merritt A. SolodkovA. Solodkov T. LeCompteT. LeCompte
MuonMuon To Be NamedTo Be Named J.F. LaporteJ.F. Laporte A. RimoldiA. Rimoldi S. GoldfarbS. Goldfarb
LVL 2 Trigger/ LVL 2 Trigger/
Trigger DAQTrigger DAQ
S. GeorgeS. George S. TapproggeS. Tapprogge M. WeilersM. Weilers A. AmorimA. Amorim
Event FilterEvent Filter F. TouchardF. Touchard M. BosmanM. Bosman
Physics Coordinator: F.Gianotti Chief Architect: D.Quarrie
November 01November 01John Huth John Huth , LHC Computing 5
International ATLAS Computing Org.
simulation reconstruction database coordinator
QC group simulation reconstruction database Arch. team
Event filter
Technical Group
National Comp. Board
Comp. Steering Group Physics
Comp. Oversight Board
Detector system
November 01November 01John Huth John Huth , LHC Computing 6
Recent Events
Progress toward coherent, integrated effortProgress toward coherent, integrated effort First Software Agreement Signed! (Control/framework)
Second one in progress (QA/AC)
Data Challenge Coordinator named (Gilbert Poulard)
Lund physics meeting (Lund Athena release)
ARC Report Endorsement of Athena
Upcoming Data Challenge 0 Upcoming Data Challenge 0 Continuity test – November Athena release
Personnel ChangesPersonnel Changes D. Malon now solo data management leader
Helge Meinhard – planning, moves to IT Division Now replaced by Torre Wenaus
November 01November 01John Huth John Huth , LHC Computing 7
Project Core SW FTE
10.5
10
643
8
6U.S.FranceU.K.CERNItalyOtherNeeded
November 01November 01John Huth John Huth , LHC Computing 8
FTE Fraction of Core SW
22%
21%
13%8%6%
17%
13% U.S.FranceU.K.CERNItalyOtherNeeded
November 01November 01John Huth John Huth , LHC Computing 9
U.S. ATLAS Goals
Deliverables to International ATLAS and LHC projectsDeliverables to International ATLAS and LHC projects Software
Control/framework (SW agreement signed) Portion of data management Event Store Collaboratory tools Detector subsystem reconstruction
Grid integration Computing resources devoted to data analysis, simulation
Tier 1, Tier 2 centers
Support of U.S. ATLAS PhysicistsSupport of U.S. ATLAS Physicists Computing resources Support functions (librarian, nightly builds, site support)
November 01November 01John Huth John Huth , LHC Computing 10
U.S. ATLAS Project
William WillisProject Manager
John HuthAssociate Project Manager,
Computing and Physics
James ShankDeputy
External Advisory Group
Ian HinchliffeManager, Physics
WBS 2.1
Torre WenausManager, Software
WBS 2.2
Bruce GibbardManager, Facilities
WBS 2.3
C. TullControl/Framework
2.2.1.1,2.2.1.2
David MalonData Management
2.2.1.3
S. RajagopalanEvent Model
2.2.1.4
J. ShankDetector Specific
2.2.2, 2.2.2.1
F. MerrittTraining
2.2.5L. Vacavant
Pixel/SCT2.2.2.2
F. LuehringTRT
2.2.2.3
S. RajagopalanLiquid Argon Calorimeter
2.2.2.4
T. LeCompteTilecal2.2.2.5
S. GoldfarbMuons2.2.2.6
S. GonzalezTrigger/DAQ
2.2.2.7
Subsystems
Core Software
R. GardnerDistributed ITInfrastructure
Facilities
TBNCollaborative
Tools2.2.3
R. BakerDeputy
T. WenausSoftware Support
Coordinator2.2.4
A. UndrusSoftware Librarian
2.2.4.1
Computing CoordinationBoard
Physics Manager, IB Convener, co-chairs
Last updated Nov 7, 2000
R. BakerTier 1 Facility
November 01November 01John Huth John Huth , LHC Computing 11
Project Management Plan
New version: extensive revisions from last yearNew version: extensive revisions from last year
Description of institutional MOU’sDescription of institutional MOU’s Two draft Inst. MOU’s exist
Liaison listLiaison list
Performance metrics establishedPerformance metrics established Personnel effort - FTE Hardware – fraction of turn-on functional per year
Change controlChange control
ReportingReporting Quarterly reports
Transition to “Research Program” in FY 07Transition to “Research Program” in FY 07
November 01November 01John Huth John Huth , LHC Computing 12
U.S. ATLAS WBS Structure
2.12.1 PhysicsPhysics Support of event generators Data challenge support in U.S.
2.2 Software2.2 Software Core Software (framework, database) Subsystem efforts Training
2.3 Facilities2.3 Facilities Tier 1 Tier 2, infrastructure (networking)
2.4 Project Management2.4 Project Management
November 01November 01John Huth John Huth , LHC Computing 13
U.S. ATLAS Developments
Athena (control/framework)Athena (control/framework) Lund release done DC 0 release Incorporation of G4 interface
DatabaseDatabase Effort augmented Coordination of Oracle,Obj, Root evaluations
FacilitiesFacilities Ramp delayed by funding profile, DC preparation reduced scope
Common grid plan worked outCommon grid plan worked out
Modest personnel ramp – BNL SW/Fac/ANLModest personnel ramp – BNL SW/Fac/ANL
Librarian support, nightly builds at BNL (from CERN)Librarian support, nightly builds at BNL (from CERN)
November 01November 01John Huth John Huth , LHC Computing 14
External Groups
iVDGL funding (Tier 2 personnel, Hardware) approvediVDGL funding (Tier 2 personnel, Hardware) approved But 50% cut in hardware relative to original planning
PPDG effort in progressPPDG effort in progress
ITR funding of Indiana (grid telemetry)ITR funding of Indiana (grid telemetry)
Integrated planning on software, facilities for gridsIntegrated planning on software, facilities for grids
Liaisons named in PMPLiaisons named in PMP GriPhyN/iVDGL – R. Gardner (J. Schopf CS liaison) PPDG – T. Wenaus (J. Schopf CS liaison) EU Data grid – C. Tull HEP Networking – S. McKee
November 01November 01John Huth John Huth , LHC Computing 15
Software
Migration from SRT to CMT begunMigration from SRT to CMT begun Effort redirection, but long term benefit
Upcoming Nov. Release of AthenaUpcoming Nov. Release of Athena Support of DC 0
Funding shortfall impactsFunding shortfall impacts Shift of D. Day (USDP, Python scripting) – postdoc hire to fill FY 03 delay hire at BNL possible – loss of ROOT expertise
Data management architecture proposed (non product specific)Data management architecture proposed (non product specific)
Root I/O service Root I/O service
G4 Integration into AthenaG4 Integration into Athena
Development of “pacman” for deployment of software (BU) at remote Development of “pacman” for deployment of software (BU) at remote
sitessites
November 01November 01John Huth John Huth , LHC Computing 16
Facilities Schedule
LHC start-up projected to be a year laterLHC start-up projected to be a year later 2005/2006 2006/2007
30% facility in 06 100% facility in 07
ATLAS Data Challenges (DC’s) have, so far, stayed fixedATLAS Data Challenges (DC’s) have, so far, stayed fixed DC0 – Nov/Dec 2001 – 105 events
Software continuity test
DC1 – Feb/Jul 2002 – 107 events ~1% scale test Data used for US ATLAS Grid testbed integration tests
DC2 – Jan/Sep 2003 – 108 events ~10% scale test A serious functionality & capacity exercise A high level of US ATLAS facilities participation is deemed very important
November 01November 01John Huth John Huth , LHC Computing 17
Facilities
Tier 1 particularly hard hit by budget shortfallTier 1 particularly hard hit by budget shortfall Delays in hiring Scalable online storage prototype work delayed approx. 7 mos. DC2 capability reduced relative to plan (1 vs. 5%) Small increments ($300k) can help substantially Year end funding of $284k from DOE (Aug 01)
Major usage of Tier 1 for shielding calculationsMajor usage of Tier 1 for shielding calculations
Anticipate major usage in DC’s and in grid testsAnticipate major usage in DC’s and in grid tests
Examination of tape vs. disk for event store at start of data takingExamination of tape vs. disk for event store at start of data taking
Tier 2Tier 2 Selection of first prototype centers (I.U., B.U.) iVDGL funding of prototype hardware Deployment of SW on testbed sites in progress
November 01November 01John Huth John Huth , LHC Computing 18
CPU Capacity (kSi95)
0
100
200
300
400
500
600
FY '03 FY '04 FY '05 FY '06 FY '07
kS
i95 2000 Profile
Current profile
November 01November 01John Huth John Huth , LHC Computing 19
US ATLAS Persistent Grid Testbed
Calren Esnet, Abilene, Nton
Esnet, Mren
UC BerkeleyLBNL-NERSC
Esnet
NPACI, Abilene
BrookhavenNationalLaboratory
Indiana University
Boston University
ArgonneNationalLaboratory
U Michigan
OklahomaUniversity
Abilene
Prototype Tier 2s
HPSS sites
November 01November 01John Huth John Huth , LHC Computing 20
Grid Efforts/Physics
Many sources of effort/sharedMany sources of effort/shared GriPhyN/iVDGL/PPDG/EU activities/New CERN mgmt.
Common U.S. ATLAS planCommon U.S. ATLAS plan Use existing tools as much as possible Use existing platforms as much as possible Gain experience in
Replica catalog Metadata description Deployment/release of tools
Philosophy is to gain expertise, not await a grand international synthesis
Physics: support hire for generator interface, data challengePhysics: support hire for generator interface, data challenge
November 01November 01John Huth John Huth , LHC Computing 21
Networking
Networking is a crucial component for the success of the Networking is a crucial component for the success of the
grid model of distributed computinggrid model of distributed computing
This has not been included as part of project funding profileThis has not been included as part of project funding profile Agency guidance
Nevertheless, it must be planned forNevertheless, it must be planned for Transatlantic planning group report (H. Newman, L. Price)
Tier 1- Tier 2 connectivity requirementsTier 1- Tier 2 connectivity requirements See talk by S. McKee
Scale and requirements are being established
Funding sources must be identified
November 01November 01John Huth John Huth , LHC Computing 22
Major Milestones
0 1 2 3 4 5 6 1 Tbyte database prototype 1/1/99 (Done)Release of Athena pre-alpha version 5/9/00 (Done)Tier 1 Processor Farm Prototype 9/29/00 (Done)Athena alpha release 9/29/00 (Done)Geant3 digi data available 10/30/00 (Done)Athena Beta Release 12/29/00 (Done)First definition of regional centers 1/1/01 (Done)Decide on database product 6/29/01 Delay * * *Tier 1 Storage Prototype 10/1/01 Part * *MDC0 Completed 12/12/01Full validation of G4 physics 12/31/01 Delay * * *MDC 1 Completed 7/30/02Computing TDR Finished 11/29/02Tier 1 Upgrade (for MDC2) 12/31/02Tier 1 Large Scale Test (MDC2) 9/30/03Physics readiness report completed 6/30/04Full software chain in real envioron. 7/30/04Full DB infrastructure available 12/31/0420% Processing Farm Prototype 9/30/05 Delay * *Tier 1 Full scale 10/2/06 Delay
November 01November 01John Huth John Huth , LHC Computing 23
Comments on Budgeting
Agencies so far have worked hard to come up the profile, but, given Agencies so far have worked hard to come up the profile, but, given
budget issues, this had been difficult, have come up shortbudget issues, this had been difficult, have come up short Construction project borrowing
We cannot plan based on our wishes, but rather realistic expectationsWe cannot plan based on our wishes, but rather realistic expectations
Current budgeting profileCurrent budgeting profile Increase in grid activities – long term benefit/short term redirection of effort Need to factor in networking costs for Tier 1-Tier 2 connections (note:
Transatlantic report) – via external funds, but must be budgeted
Relief in form of overall NSF funding for “research program”Relief in form of overall NSF funding for “research program” Overall profile for M+O, upgrades and computing Major components in computing:
Tier 2 sites, grid integration with ATLAS software SW Professionals located at CERN (hires through universities)
November 01November 01John Huth John Huth , LHC Computing 24
Funding Guidance
Project Funding Sources
02000400060008000
100001200014000160001800020000
FY 02 FY 03 FY 04 FY 05 FY 06
Fiscal Year
At
year
$k
Grid NSF
Non-Grid NSF
DOE
November 01November 01John Huth John Huth , LHC Computing 25
Budget Profile by Item
0
5000
10000
15000
20000
25000
FY 02 FY 03 FY 04 FY 05 FY 06 FY 07
At
Yea
r $K
2.4 Reserve
2.4 Support
2.1 Physics
2.3.2 Dist. IT
2.3.1 Tier 1
2.2 Software
November 01November 01John Huth John Huth , LHC Computing 26
WBS Number Description FY 02 FY 03 FY 04 FY05 FY06 Total FY07
2 US Atlas Computing 3,581 5,328 8,201 10,123 14,457 41,690 17,7552.1 Physics 100 147 196 210 215 868 2152.2 Software Projects 2252 2400 3043 3446 3547 14688 35002.3 Computing Facilities2.3.1 Tier 1 839 1701 3392 4467 7575 17974 106152.3.2 Distributed IT 290 780 1120 1850 2970 7010 32652.9 Project Support 100 300 450 150 150 1150 160
Management Reserve 0 250 820 1,012 1,446 3528 1,776
US ATLAS Computing w/reserve 3,581 5,578 9,021 11,135 15,903 45,218 19,531
AY k$'sFiscal Years
November 01November 01John Huth John Huth , LHC Computing 27
FTE Profile
0
10
20
30
40
50
60
FY 02 FY 03 FY 04 FY 05 FY 06 FY 07
FT
E
2.4 Support2.3.2 Dist. IT2.3.1 Tier 12.2 Software2.1 Physics
November 01November 01John Huth John Huth , LHC Computing 28
FTE by Category in 02
0.75
11.3
4.4
11.2
2.1 Physics
2.2 Software
2.3.1 Tier 1
2.3.2 Dist. IT
2.4 Support
November 01November 01John Huth John Huth , LHC Computing 29
FTE’s in 07
113.5
25
101.2
2.1 Physics2.2 Software2.3.1 Tier 12.3.2 Dist. IT2.4 Support
November 01November 01John Huth John Huth , LHC Computing 30
Matching of Profiles
0
5000
10000
15000
20000
25000
FY 02 FY 03 FY 04 FY 05 FY 06 FY 07
ProfileGuidance
November 01November 01John Huth John Huth , LHC Computing 31
Risks
SoftwareSoftware Loss of expertise in control/framework (scripting)
New hire as mitigation
Loss of Ed Frank (U. Chicago) – data management Delay of new hire at BNL – ROOT persistency expertise Support questions in tools (e.g. CMT)
FacilitiesFacilities Slowed ramp-up in personnel, hardware Facility preparation for DC2 implies reduced scale
GridGrid Some shift of effort from core areas into grid developments –
following the development of integrated model of comp. centers
November 01November 01John Huth John Huth , LHC Computing 32
NSF Proposal
Covers Computing, upgrades and M+OCovers Computing, upgrades and M+O
Computing:Computing: 3 Software FTE’s – located at CERN (hire by University)
Alleviates shortfall of 2 FTE, covers remaining portion in out years
Physics support person Support of generator interfaces, data challenges
Main source of Tier 2 funding Tier 2 hardware Personnel Common (w/ US CMS) team to debug last mile networking problems
Alleviates shortfalls in the programAlleviates shortfalls in the program N.B. this also frees up DOE funds for labs, allowing a better Tier 1 ramp,
preparation for data challenges.
November 01November 01John Huth John Huth , LHC Computing 33
FY 02 FY 03 FY 04 FY 05 FY 06
Physics 124 150 200 200 200Software 820 844 870 896 925Tier 2Local staff 542 620 653 697 930Central staff 0 0 0 300 465Hardware 240 240 870 1870 2000
Reserve 259 278 389 594 678
Sum 1985 2132 2982 4257 4733
Related ProjectsiVDGL 403 532 550 449 457GriPhyN 139 139 139 0 0Grid Telemetry 167 167 167
Sum Related 709 838 856 449 457
Total 2694 2970 3838 4706 5190
November 01November 01John Huth John Huth , LHC Computing 34
Summary
Much progress on many frontsMuch progress on many fronts
Funding profile is still an issueFunding profile is still an issue Personnel ramp Ramp of facility Some possible solutions
Funds for loan payback are incremental “Research Program” NSF proposal is necessary
Progress in International collaborationProgress in International collaboration Software agreements
The single biggest help from the committee would be a favorable The single biggest help from the committee would be a favorable
recommendation on the computing request in the NSF proposalrecommendation on the computing request in the NSF proposal
In addition, endorsement of proposed project scope, schedule, budgets In addition, endorsement of proposed project scope, schedule, budgets
and management plan.and management plan.