Software Status/Plans Torre Wenaus, BNL/CERN US ATLAS Software Manager US ATLAS PCAP Review November 14, 2002
Dec 22, 2015
Software Status/Plans
Torre Wenaus, BNL/CERN
US ATLAS Software Manager
US ATLAS PCAP Review
November 14, 2002
PCAP Review, November 14, 2002 Slide 2 Torre Wenaus, BNL/CERN
U.S. ATLAS Software Project Overview
Control framework and architecture Chief Architect, principal development role, ATLAS LCG
applications area liaison Databases and data management
Database Leader, primary ATLAS expertise on ROOT/relational baseline
Software support for development and analysis Software librarian, quality control, software development tools,
training… Automated build/testing system adopted by Int’l ATLAS
Subsystem software roles complementing hardware responsibilities Muon system software coordinator
Scope commensurate with U.S. in ATLAS: ~20% of overall effort Commensurate representation on steering group
Strong role and participation in LCG common effort
Recent developments in green
PCAP Review, November 14, 2002 Slide 3 Torre Wenaus, BNL/CERN
U.S. ATLAS Software Organization
William WillisProject Manager
John HuthAssociate Project Manager,
Computing and Phy sics WBS 2
James ShankDeputy
External Advisory Group
Ian HinchliffeManager, Phy sics
WBS 2.1
Torre WenausManager, Sof tware
WBS 2.2
Bruce GibbardManager, Facilities
WBS 2.3
D. QuarrieControl/Framework
2.2.1.1,2.2.1.2
David MalonData Management
2.2.1.3
S. RajagopalanEv ent Model
2.2.1.4
J. ShankDetector Specif ic
2.2.2, 2.2.2.1
F. MerrittTraining
2.2.5L. Vacavant
Pixel/SCT2.2.2.2
F. LuehringTRT
2.2.2.3
S. RajagopalanLiquid Argon Calorimeter
2.2.2.4
T. LeCompteTilecal2.2.2.5
S. GoldfarbMuons2.2.2.6
S. GonzalezTrigger/DAQ
2.2.2.7
Subsy stems
Core Sof tware
R. BakerTier 1 Facility
R. GardnerDistributed ITInf rastructure
Facilities
TBNCollaborativ e
Tools2.2.3
R. BakerDeputy
T. WenausSof tware Support
Coordinator2.2.4
A. UndrusSof tware Librarian
2.2.4.1
Computing CoordinationBoard
Phy sics Manager, IB Conv ener, co-chairs
PCAP Review, November 14, 2002 Slide 4 Torre Wenaus, BNL/CERN
U.S. ATLAS - ATLAS Coordination
US roles in Int’l ATLAS software:US roles in Int’l ATLAS software:
D. Quarrie (LBNL), Chief ArchitectD. Quarrie (LBNL), Chief Architect
D. Malon (ANL), Database CoordinatorD. Malon (ANL), Database Coordinator
P. Nevski (BNL), Geant3 Simulation P. Nevski (BNL), Geant3 Simulation Coordinator, Simulation production leadCoordinator, Simulation production lead
C. Tull (LBNL), EDG WP8 LiaisonC. Tull (LBNL), EDG WP8 Liaison
H. Ma (BNL), Raw Data CoordinatorH. Ma (BNL), Raw Data Coordinator
T. Wenaus (BNL), Planning OfficerT. Wenaus (BNL), Planning Officer
US International
See task matrix
William WillisU.S. ATLAS Project
Manager
Peter JenniATLAS
Spokesperson
John HuthAssociate PM
NormanMcCubbinSoftware
Coordiinator
I. HinchliffePhysics SM
F. GianottiPhysics
Coordinator
T. WenausSoftware SM
B. GibbardFacilities SM
D. QuarrieFramework
D. MalonDatabase
SubsystemSoftware
T. WenausPlanning Officer
D. QuarrieChief Archectect
D.MalonDatabase
A. PutzerNCB
L. PeriniATLAS GRID
R. GardnerDistributed Computing
SubsystemSoftware
PCAP Review, November 14, 2002 Slide 5 Torre Wenaus, BNL/CERN
ATLAS Subsystem/Task Matrix
Offline
Coordinator
Reconstruction Simulation Database
Chair N. McCubbin D. Rousseau A. Dell’Acqua D. Malon
Inner Detector D. Barberis D. Rousseau F. Luehring S. Bentvelsen /
D. Calvet
Liquid Argon J. Collot S. Rajagopalan M. Leltchouk H. Ma
Tile Calorimeter A. Solodkov F. Merritt V.Tsulaya T. LeCompte
Muon J.Shank J.F. Laporte A. Rimoldi S. Goldfarb
LVL 2 Trigger/
Trigger DAQ
S. George S. Tapprogge M. Weilers A. Amorim /
F. Touchard
Event Filter V. Vercesi F. Touchard
Computing Steering Group members/attendees: 4 of 19 Computing Steering Group members/attendees: 4 of 19 from US (Malon, Quarrie, Shank, Wenaus)from US (Malon, Quarrie, Shank, Wenaus)
Physics Coordinator: F.Gianotti
Chief Architect: D.Quarrie
PCAP Review, November 14, 2002 Slide 6 Torre Wenaus, BNL/CERN
Project Planning Status
U.S./Int’l ATLAS WBS/PBS and schedule fully unified US/Int’l software planning covered by same person (TW)
Synergies outweigh the added burden of the ATLAS Planning Officer role
No ‘coordination layer’ between US and Int’l ATLAS planning
Possible because of how the ATLAS Planning Officer role is currently scoped
As pointed out by an informal ATLAS computing review in March, ATLAS would benefit from a full FTE Planning Officer
I have a standing offer to the Computing Coordinator: to step aside if/when a capable person with more time is found
Until then, I scope the job to what I have time for and what is highest priority
PCAP Review, November 14, 2002 Slide 7 Torre Wenaus, BNL/CERN
ATLAS Computing Planning
US led a comprehensive review and update of ATLAS computing schedule in the spring
Milestone count increased by 50% to 600; many others updated Milestones and planning coordinated around DC schedule Reasonably comprehensive and detailed through 2002
New round underway now to flesh out 2003 schedule US core activity scheduling in reasonable shape
Long term milestones reworked to reflect LHC schedule, LCG Centered around escalating data challenges
Weak decision making, still a problem, translates to (among other things) weak planning
Strong recommendation of the March review to fix this Should be fixed in the present computing management
reorganization
PCAP Review, November 14, 2002 Slide 8 Torre Wenaus, BNL/CERN
Summary Major Milestones
0 1 2 3 4 5 6 1 Tbyte database prototype (Done)Release of Athena pre-alpha version (Done)Athena alpha release (Done)Geant3 digi data available (Done)Athena beta release (Done)Athena accepted (ARC concluded) (Done)Athena Lund release (Done)Event store architecture design document (Done)DC0 production release (Done)Decide on database product (Done)DC0 Completed - continuity test (Done)Full validation of Geant4 physics DelayDC1 Completed DelayComputing TDR Finished (Align with LCG) DelayDC2 Completed (followed by annual DCs) DelayDC3 Completed (Exercise LCG-3) NewPhysics readiness report completed DelayDC4 Completed (Align with wedge test) NewFull chain in real environment (DC5) Delay
Green: Done Gray: Original date Blue: Current date
PCAP Review, November 14, 2002 Slide 9 Torre Wenaus, BNL/CERN
Major Milestones
One DC per year until startup
PCAP Review, November 14, 2002 Slide 10 Torre Wenaus, BNL/CERN
Data Challenge 1
DC1 phase 1 (simu production for HLT TDR) executed successfully World-wide operation
Phase 2, starting in the next weeks, focuses on testing of new software Introduction and testing of new event data model Intensive Geant4 usage Intensive Grid usage Data production for Physics and computing model studies First tests of computing model Analysis using Analysis Object Data (AOD)
DC1 Phase 2 too early for LCG common persistency POOL production use
POOL production releases in spring/summer 2003
PCAP Review, November 14, 2002 Slide 11 Torre Wenaus, BNL/CERN
Software Support, Quality Control
New releases are available in the US typically ~1-2 days after CERN Provided in AFS for use throughout the US
Librarian receives help requests and queries from ~25 people in the US US-developed nightly build facility used throughout ATLAS
Central tool in the day to day work of developers and the release process Recently expanded as framework for progressively integrating more quality
control and testing Testing at component, package and application level Code checking to be integrated
CERN support functions being transferred to new ATLAS librarian BNL-based nightlies recently resumed
Much more stable build environment than CERN at the moment Use timely, robust nightlies to promote usage of the Tier 1 for development
PCAP Review, November 14, 2002 Slide 12 Torre Wenaus, BNL/CERN
Software Support, Quality Control (2)
Testing integrated into automated builds Unit tests, package tests, integration/system tests
ATLAS has (finally!) established a dedicated support team (SIT) for software infrastructure, testing, release management etc.
U.S. represented by the U.S. ATLAS librarian, an active team member
SIT needs a dedicated leader (currently the rotating, and overloaded, release manager heads SIT)
Provides a much needed context for U.S. support and QA efforts pacman (Boston U) for remote software installation
Adopted by grid projects for VDT, and a central tool in US grid testbed work
PCAP Review, November 14, 2002 Slide 13 Torre Wenaus, BNL/CERN
Grid Software
Software development within the ATLAS complements of the grid projects is being managed as an integral part of the software effort
Grid software activities tightly integrated into ongoing core software program, for maximal relevance and return
Grid project programs consistent with this have been developed And has been successful
e.g. Distributed data manager tool (Magda) we developed was adopted ATLAS-wide for data management in the DCs
Grid goals, schedules integrated with ATLAS (particularly DC) program
However we do suffer some program distortion e.g. we have to limit effort on providing ATLAS with event storage
capability in order to do work on longer-range, higher-level distributed data management services
PCAP Review, November 14, 2002 Slide 14 Torre Wenaus, BNL/CERN
Effort Level Changes
ANL/Chicago – loss of .5 FTE in DB Ed Frank departure; no resources to replace Another .5 FTE recently lost, will be replaced
BNL – cancelled 1 FTE new hire in data management Insufficient funding in the project and the base program to sustain the bare-bones
plan Results in transfer of DB effort to grid (PPDG) effort – because the latter pays the
bills, even if it distorts our program towards lesser priorities LBNL – stable project-supported FTE count in architecture/framework
But loss of base support is threatening effort level and deliverables Grid funding being sought to ameliorate
DB effort hard-hit, but ameliorated by common project Because the work is now in the context of a broad common project, US can still
sustain our major role in ATLAS DB A material example of common effort translating into savings (even if we wouldn’t
have chosen to structure the savings this way!)
PCAP Review, November 14, 2002 Slide 15 Torre Wenaus, BNL/CERN
Personnel Priorities for FY02, FY03
This is how we are doing relative to goals… Sustain LBNL (4.5FTE) and ANL (3FTE) support
This we are doing so far. Add FY02, FY03 1FTE increments at BNL to reach 3FTEs
Failed in 02; BNL hire cancelled. Should recover to 3 FTEs in FY03 Restore the .5FTE lost at UC to ANL
No resources Establish sustained presence at CERN.
No resources, despite being a very high priority We rely on labs to continue base program and other lab support to
sustain existing complement of developers And needed base program support is not there. Lab base programs
are being hammered.
PCAP Review, November 14, 2002 Slide 16 Torre Wenaus, BNL/CERN
General and longer term priorities
These are reflected in the software request in the research program proposal (and go back as far as our original Jan 2000 project plan)
Priorities in order: Sustain existing ANL, BNL, LBNL efforts Complete the ramp of the lab based core developer FTEs to the
long-planned levels ANL 3.5 FTEs, LBNL 4.5 FTEs, BNL 4 FTEs
Establish, over and above these lab levels, presence at CERN of at least 2 core developer FTEs
In addition to any lab people who might be located at CERN Establish effort at the core-subsystem interface – sited mainly at
universities and possibly CERN – to support the translation of core developments into established
software employed by end users better support the leadership roles held by the US with developer
effort capable of translating decisions into established solutions
PCAP Review, November 14, 2002 Slide 17 Torre Wenaus, BNL/CERN
SW Funding Profile Comparisons
0.00
1.00
2.00
3.00
4.00
5.00
6.00
FY01 FY02 FY03 FY04 FY05 FY06
M$
2000 agency guideline
January 2000 PMP
11/2001 guideline
‘Compromise profile’
requested in 2000
Mid 02 bare bones
PCAP Review, November 14, 2002 Slide 18 Torre Wenaus, BNL/CERN
Concluding Remarks
No strategic changes; program is working, but stressed by funding
US has consolidated the leading roles in our targeted core software areas
Involved with new LCG common efforts in all our core areas
Architecture/framework effort level being sustained so far
And is delivering the baseline core software of ATLAS
Database/data mgmt effort reduced but so far preserving our key technical expertise
Leveraging that expertise for a strong role in common project
Cannot tolerate further reduction in a key strategic US core area
US major contributor to software infrastructure and QA in ATLAS
Recent emphasis: improve QA, and make the US development and production environment as effective as possible
Soft support from the project and base programs while the emphasis on grids grows is distorting our program in a troubling way