U.S. ATLAS Project Overview John Huth Harvard University LHC Computing Review FNAL November 2001.

U.S. ATLAS Project Overview

John HuthJohn Huth

Harvard UniversityHarvard University

LHC Computing ReviewLHC Computing Review

FNALFNALNovember 2001November 2001

November 01November 01John Huth John Huth , LHC Computing 2

Outline

International and U.S. ATLASInternational and U.S. ATLAS Organization U.S. ATLAS External Groups

Project Management PlanProject Management Plan

MilestonesMilestones

StatusStatus Software Facilities Grid efforts Physics

Funding ProfileFunding Profile Impact


International ATLAS

Computing Oversight BoardComputing Oversight Board

Computing Steering GroupComputing Steering Group Matrix of detector/task orientation

PBS structure gives tasks, schedules, resource loadingPBS structure gives tasks, schedules, resource loading Maps directly onto U.S. ATLAS WBS

Planning officer is now Torre Wenaus (U.S. ATLAS SW Mgr.)

Software deliverables delineated in Software AgreementsSoftware deliverables delineated in Software Agreements

Major milestones associated with releases, data challengesMajor milestones associated with releases, data challenges

Data Challenge coordinator: Gilbert Poulard (CERN)Data Challenge coordinator: Gilbert Poulard (CERN)


ATLAS Detector/Task matrix

Offline Offline

CoordinatorCoordinator

ReconstructionReconstruction SimulationSimulation DatabaseDatabase

ChairChair N. McCubbinN. McCubbin D. RousseauD. Rousseau K. AmakoK. Amako D. MalonD. Malon

Inner DetectorInner Detector D. BarberisD. Barberis D. RousseauD. Rousseau F. LuehringF. Luehring S. BentvelsenS. Bentvelsen

Liquid ArgonLiquid Argon J. CollotJ. Collot S. RajagopalanS. Rajagopalan M. LeltchoukM. Leltchouk S. SimionS. Simion//

R. SobieR. Sobie

Tile CalorimeterTile Calorimeter A. SolodkovA. Solodkov F. MerrittF. Merritt A. SolodkovA. Solodkov T. LeCompteT. LeCompte

MuonMuon To Be NamedTo Be Named J.F. LaporteJ.F. Laporte A. RimoldiA. Rimoldi S. GoldfarbS. Goldfarb

LVL 2 Trigger/ LVL 2 Trigger/

Trigger DAQTrigger DAQ

S. GeorgeS. George S. TapproggeS. Tapprogge M. WeilersM. Weilers A. AmorimA. Amorim

Event FilterEvent Filter F. TouchardF. Touchard M. BosmanM. Bosman

Physics Coordinator: F.Gianotti Chief Architect: D.Quarrie


International ATLAS Computing Org.

simulation reconstruction database coordinator

QC group simulation reconstruction database Arch. team

Event filter

Technical Group

National Comp. Board

Comp. Steering Group Physics

Comp. Oversight Board

Detector system


Recent Events

Progress toward coherent, integrated effortProgress toward coherent, integrated effort First Software Agreement Signed! (Control/framework)

Second one in progress (QA/AC)

Data Challenge Coordinator named (Gilbert Poulard)

Lund physics meeting (Lund Athena release)

ARC Report Endorsement of Athena

Upcoming Data Challenge 0 Upcoming Data Challenge 0 Continuity test – November Athena release

Personnel ChangesPersonnel Changes D. Malon now solo data management leader

Helge Meinhard – planning, moves to IT Division Now replaced by Torre Wenaus


Project Core SW FTE

10.5

10

643

8

6U.S.FranceU.K.CERNItalyOtherNeeded


FTE Fraction of Core SW

22%

21%

13%8%6%

17%

13% U.S.FranceU.K.CERNItalyOtherNeeded


U.S. ATLAS Goals

Deliverables to International ATLAS and LHC projectsDeliverables to International ATLAS and LHC projects Software

Control/framework (SW agreement signed) Portion of data management Event Store Collaboratory tools Detector subsystem reconstruction

Grid integration Computing resources devoted to data analysis, simulation

Tier 1, Tier 2 centers

Support of U.S. ATLAS PhysicistsSupport of U.S. ATLAS Physicists Computing resources Support functions (librarian, nightly builds, site support)


U.S. ATLAS Project

William WillisProject Manager

John HuthAssociate Project Manager,

Computing and Physics

James ShankDeputy

External Advisory Group

Ian HinchliffeManager, Physics

WBS 2.1

Torre WenausManager, Software

WBS 2.2

Bruce GibbardManager, Facilities

WBS 2.3

C. TullControl/Framework

2.2.1.1,2.2.1.2

David MalonData Management

2.2.1.3

S. RajagopalanEvent Model

2.2.1.4

J. ShankDetector Specific

2.2.2, 2.2.2.1

F. MerrittTraining

2.2.5L. Vacavant

Pixel/SCT2.2.2.2

F. LuehringTRT

2.2.2.3

S. RajagopalanLiquid Argon Calorimeter

2.2.2.4

T. LeCompteTilecal2.2.2.5

S. GoldfarbMuons2.2.2.6

S. GonzalezTrigger/DAQ

2.2.2.7

Subsystems

Core Software

R. GardnerDistributed ITInfrastructure

Facilities

TBNCollaborative

Tools2.2.3

R. BakerDeputy

T. WenausSoftware Support

Coordinator2.2.4

A. UndrusSoftware Librarian

2.2.4.1

Computing CoordinationBoard

Physics Manager, IB Convener, co-chairs

Last updated Nov 7, 2000

R. BakerTier 1 Facility


Project Management Plan

New version: extensive revisions from last yearNew version: extensive revisions from last year

Description of institutional MOU’sDescription of institutional MOU’s Two draft Inst. MOU’s exist

Liaison listLiaison list

Performance metrics establishedPerformance metrics established Personnel effort - FTE Hardware – fraction of turn-on functional per year

Change controlChange control

ReportingReporting Quarterly reports

Transition to “Research Program” in FY 07Transition to “Research Program” in FY 07


U.S. ATLAS WBS Structure

2.12.1 PhysicsPhysics Support of event generators Data challenge support in U.S.

2.2 Software2.2 Software Core Software (framework, database) Subsystem efforts Training

2.3 Facilities2.3 Facilities Tier 1 Tier 2, infrastructure (networking)

2.4 Project Management2.4 Project Management


U.S. ATLAS Developments

Athena (control/framework)Athena (control/framework) Lund release done DC 0 release Incorporation of G4 interface

DatabaseDatabase Effort augmented Coordination of Oracle,Obj, Root evaluations

FacilitiesFacilities Ramp delayed by funding profile, DC preparation reduced scope

Common grid plan worked outCommon grid plan worked out

Modest personnel ramp – BNL SW/Fac/ANLModest personnel ramp – BNL SW/Fac/ANL

Librarian support, nightly builds at BNL (from CERN)Librarian support, nightly builds at BNL (from CERN)


External Groups

iVDGL funding (Tier 2 personnel, Hardware) approvediVDGL funding (Tier 2 personnel, Hardware) approved But 50% cut in hardware relative to original planning

PPDG effort in progressPPDG effort in progress

ITR funding of Indiana (grid telemetry)ITR funding of Indiana (grid telemetry)

Integrated planning on software, facilities for gridsIntegrated planning on software, facilities for grids

Liaisons named in PMPLiaisons named in PMP GriPhyN/iVDGL – R. Gardner (J. Schopf CS liaison) PPDG – T. Wenaus (J. Schopf CS liaison) EU Data grid – C. Tull HEP Networking – S. McKee


Software

Migration from SRT to CMT begunMigration from SRT to CMT begun Effort redirection, but long term benefit

Upcoming Nov. Release of AthenaUpcoming Nov. Release of Athena Support of DC 0

Funding shortfall impactsFunding shortfall impacts Shift of D. Day (USDP, Python scripting) – postdoc hire to fill FY 03 delay hire at BNL possible – loss of ROOT expertise

Data management architecture proposed (non product specific)Data management architecture proposed (non product specific)

Root I/O service Root I/O service

G4 Integration into AthenaG4 Integration into Athena

Development of “pacman” for deployment of software (BU) at remote Development of “pacman” for deployment of software (BU) at remote

sitessites


Facilities Schedule

LHC start-up projected to be a year laterLHC start-up projected to be a year later 2005/2006 2006/2007

30% facility in 06 100% facility in 07

ATLAS Data Challenges (DC’s) have, so far, stayed fixedATLAS Data Challenges (DC’s) have, so far, stayed fixed DC0 – Nov/Dec 2001 – 105 events

Software continuity test

DC1 – Feb/Jul 2002 – 107 events ~1% scale test Data used for US ATLAS Grid testbed integration tests

DC2 – Jan/Sep 2003 – 108 events ~10% scale test A serious functionality & capacity exercise A high level of US ATLAS facilities participation is deemed very important


Facilities

Tier 1 particularly hard hit by budget shortfallTier 1 particularly hard hit by budget shortfall Delays in hiring Scalable online storage prototype work delayed approx. 7 mos. DC2 capability reduced relative to plan (1 vs. 5%) Small increments ($300k) can help substantially Year end funding of $284k from DOE (Aug 01)

Major usage of Tier 1 for shielding calculationsMajor usage of Tier 1 for shielding calculations

Anticipate major usage in DC’s and in grid testsAnticipate major usage in DC’s and in grid tests

Examination of tape vs. disk for event store at start of data takingExamination of tape vs. disk for event store at start of data taking

Tier 2Tier 2 Selection of first prototype centers (I.U., B.U.) iVDGL funding of prototype hardware Deployment of SW on testbed sites in progress


CPU Capacity (kSi95)

0

100

200

300

400

500

600

FY '03 FY '04 FY '05 FY '06 FY '07

kS

i95 2000 Profile

Current profile


US ATLAS Persistent Grid Testbed

Calren Esnet, Abilene, Nton

Esnet, Mren

UC BerkeleyLBNL-NERSC

Esnet

NPACI, Abilene

BrookhavenNationalLaboratory

Indiana University

Boston University

ArgonneNationalLaboratory

U Michigan

OklahomaUniversity

Abilene

Prototype Tier 2s

HPSS sites


Grid Efforts/Physics

Many sources of effort/sharedMany sources of effort/shared GriPhyN/iVDGL/PPDG/EU activities/New CERN mgmt.

Common U.S. ATLAS planCommon U.S. ATLAS plan Use existing tools as much as possible Use existing platforms as much as possible Gain experience in

Replica catalog Metadata description Deployment/release of tools

Philosophy is to gain expertise, not await a grand international synthesis

Physics: support hire for generator interface, data challengePhysics: support hire for generator interface, data challenge


Networking

Networking is a crucial component for the success of the Networking is a crucial component for the success of the

grid model of distributed computinggrid model of distributed computing

This has not been included as part of project funding profileThis has not been included as part of project funding profile Agency guidance

Nevertheless, it must be planned forNevertheless, it must be planned for Transatlantic planning group report (H. Newman, L. Price)

Tier 1- Tier 2 connectivity requirementsTier 1- Tier 2 connectivity requirements See talk by S. McKee

Scale and requirements are being established

Funding sources must be identified


Major Milestones

0 1 2 3 4 5 6 1 Tbyte database prototype 1/1/99 (Done)Release of Athena pre-alpha version 5/9/00 (Done)Tier 1 Processor Farm Prototype 9/29/00 (Done)Athena alpha release 9/29/00 (Done)Geant3 digi data available 10/30/00 (Done)Athena Beta Release 12/29/00 (Done)First definition of regional centers 1/1/01 (Done)Decide on database product 6/29/01 Delay * * *Tier 1 Storage Prototype 10/1/01 Part * *MDC0 Completed 12/12/01Full validation of G4 physics 12/31/01 Delay * * *MDC 1 Completed 7/30/02Computing TDR Finished 11/29/02Tier 1 Upgrade (for MDC2) 12/31/02Tier 1 Large Scale Test (MDC2) 9/30/03Physics readiness report completed 6/30/04Full software chain in real envioron. 7/30/04Full DB infrastructure available 12/31/0420% Processing Farm Prototype 9/30/05 Delay * *Tier 1 Full scale 10/2/06 Delay


Comments on Budgeting

Agencies so far have worked hard to come up the profile, but, given Agencies so far have worked hard to come up the profile, but, given

budget issues, this had been difficult, have come up shortbudget issues, this had been difficult, have come up short Construction project borrowing

We cannot plan based on our wishes, but rather realistic expectationsWe cannot plan based on our wishes, but rather realistic expectations

Current budgeting profileCurrent budgeting profile Increase in grid activities – long term benefit/short term redirection of effort Need to factor in networking costs for Tier 1-Tier 2 connections (note:

Transatlantic report) – via external funds, but must be budgeted

Relief in form of overall NSF funding for “research program”Relief in form of overall NSF funding for “research program” Overall profile for M+O, upgrades and computing Major components in computing:

Tier 2 sites, grid integration with ATLAS software SW Professionals located at CERN (hires through universities)


Funding Guidance

Project Funding Sources

02000400060008000

100001200014000160001800020000

FY 02 FY 03 FY 04 FY 05 FY 06

Fiscal Year

At

year

$k

Grid NSF

Non-Grid NSF

DOE


Budget Profile by Item

0

5000

10000

15000

20000

25000

FY 02 FY 03 FY 04 FY 05 FY 06 FY 07

At

Yea

r $K

2.4 Reserve

2.4 Support

2.1 Physics

2.3.2 Dist. IT

2.3.1 Tier 1

2.2 Software


WBS Number Description FY 02 FY 03 FY 04 FY05 FY06 Total FY07

2 US Atlas Computing 3,581 5,328 8,201 10,123 14,457 41,690 17,7552.1 Physics 100 147 196 210 215 868 2152.2 Software Projects 2252 2400 3043 3446 3547 14688 35002.3 Computing Facilities2.3.1 Tier 1 839 1701 3392 4467 7575 17974 106152.3.2 Distributed IT 290 780 1120 1850 2970 7010 32652.9 Project Support 100 300 450 150 150 1150 160

Management Reserve 0 250 820 1,012 1,446 3528 1,776

US ATLAS Computing w/reserve 3,581 5,578 9,021 11,135 15,903 45,218 19,531

AY k$'sFiscal Years


FTE Profile

0

10

20

30

40

50

60

FY 02 FY 03 FY 04 FY 05 FY 06 FY 07

FT

E

2.4 Support2.3.2 Dist. IT2.3.1 Tier 12.2 Software2.1 Physics


FTE by Category in 02

0.75

11.3

4.4

11.2

2.1 Physics

2.2 Software

2.3.1 Tier 1

2.3.2 Dist. IT

2.4 Support


FTE’s in 07

113.5

25

101.2

2.1 Physics2.2 Software2.3.1 Tier 12.3.2 Dist. IT2.4 Support


Matching of Profiles

0

5000

10000

15000

20000

25000

FY 02 FY 03 FY 04 FY 05 FY 06 FY 07

ProfileGuidance


Risks

SoftwareSoftware Loss of expertise in control/framework (scripting)

New hire as mitigation

Loss of Ed Frank (U. Chicago) – data management Delay of new hire at BNL – ROOT persistency expertise Support questions in tools (e.g. CMT)

FacilitiesFacilities Slowed ramp-up in personnel, hardware Facility preparation for DC2 implies reduced scale

GridGrid Some shift of effort from core areas into grid developments –

following the development of integrated model of comp. centers


NSF Proposal

Covers Computing, upgrades and M+OCovers Computing, upgrades and M+O

Computing:Computing: 3 Software FTE’s – located at CERN (hire by University)

Alleviates shortfall of 2 FTE, covers remaining portion in out years

Physics support person Support of generator interfaces, data challenges

Main source of Tier 2 funding Tier 2 hardware Personnel Common (w/ US CMS) team to debug last mile networking problems

Alleviates shortfalls in the programAlleviates shortfalls in the program N.B. this also frees up DOE funds for labs, allowing a better Tier 1 ramp,

preparation for data challenges.


FY 02 FY 03 FY 04 FY 05 FY 06

Physics 124 150 200 200 200Software 820 844 870 896 925Tier 2Local staff 542 620 653 697 930Central staff 0 0 0 300 465Hardware 240 240 870 1870 2000

Reserve 259 278 389 594 678

Sum 1985 2132 2982 4257 4733

Related ProjectsiVDGL 403 532 550 449 457GriPhyN 139 139 139 0 0Grid Telemetry 167 167 167

Sum Related 709 838 856 449 457

Total 2694 2970 3838 4706 5190


Summary

Much progress on many frontsMuch progress on many fronts

Funding profile is still an issueFunding profile is still an issue Personnel ramp Ramp of facility Some possible solutions

Funds for loan payback are incremental “Research Program” NSF proposal is necessary

Progress in International collaborationProgress in International collaboration Software agreements

The single biggest help from the committee would be a favorable The single biggest help from the committee would be a favorable

recommendation on the computing request in the NSF proposalrecommendation on the computing request in the NSF proposal

In addition, endorsement of proposed project scope, schedule, budgets In addition, endorsement of proposed project scope, schedule, budgets

and management plan.and management plan.

U.S. ATLAS Project Overview John Huth Harvard University LHC Computing Review FNAL November 2001.

Documents

lbnl pcap reviewu

atlas goalsdeliverables

atlas projectdavid

atlas sw mgr

atlas wbs structure2

atlas wbsplanning officer

data analysis

software deliverables