K. Harrison Liverpool, 14th September 2004 GANGA DEVELOPMENT - Objectives and organisation - Ganga in ATLAS and LHCb - Release status - Work in progress.

K. HarrisonLiverpool, 14th September 2004

GANGA DEVELOPMENT

- Objectives and organisation- Ganga in ATLAS and LHCb- Release status- Work in progress- Conclusions

14th September 2004 2

Main objectives

- Aim to help physicists in ATLAS and LHCb carry out studies in the complex computing environment of LHC Help configure applications developed within the Gaudi/Athena framework shared by ATLAS and LHCb Deal with submitting and monitoring jobs to distributed (Grid) and local batch systems Help users to keep track of what they’ve done Hide all technicalities: allow user to concentrate on the physics Provide a single user-friendly environment for all tasks

Ganga: Gaudi/Athena and Grid Alliance- Ganga is the Hindi name for the Ganges, and is the river goddess who carried lost souls to salvation- Ganga software will carry jobs to the Grid underworld, and hopefully bring them back


Ganga: first-level decomposition

GANGA

UI

Grid Services

MDSvc

MetadataService

WorkLoadManager

SEFile

catalog

WLM ProSvc Monitor

Internal Model

ProfileService

GAUDIProgram

Instr.…

CE


People

- Ganga is being developed as an ATLAS/LHCb common project, with support in UK from GridPP

- Current main contributors are: Development: K.Harrison, J.Moscicki, A.Soroko, C.L.Tan Technical direction: D.L.Adams (ATLAS), U.Egede (LHCb) GridPP coordination: R.W.L.Jones (ATLAS), G.N.Patrick (LHCb)

- Additional contributions, past and present, from many others: N.Brook, P.Charpentier, K.Ciba, C.Collins-Tooth, F.Harris, A.Maier, J.Martyniak, P.Mato, W.Lavrijsen, G.Rybkine, C.E.Tull Appologies to anyone left out


Organisation

- Ganga-related information regularly updated on project web site: http://ganga.web.cern.ch/ganga/- Software status is tracked using Savannah: https://savannah.cern.ch/projects/ganga/ - Have two active mailing lists, with archives accessible via web site: General interest: [email protected] Technical issues: [email protected] Usually have a telephone meeting once a week Times, agendas and minutes placed on web site and circulated to [email protected] mailing list- Give status reports at ATLAS Software Workshops, at LHCb Software Weeks, at GridPP meetings, and at various other meetings

http://ganga.web.cern.ch/ganga/









https://savannah.cern.ch/projects/ganga/







mailto:[email protected]














ATLAS and LHCb: similarities and differences (1)

- Two experiments use (almost) the same software framework: Gaudi/Athena User chooses alogrithms to run, and sets algorithm parameters, via job options LHCb uses text job options ATLAS starting to use Python job options


Gaudi architecture

Converter

Algorithm

Event DataService

PersistencyService

DataFiles

AlgorithmAlgorithm

Transient Event Store

Detec. DataService

PersistencyService

DataFiles

Transient Detector

Store

MessageService

JobOptionsService

Particle Prop.Service

OtherServices

HistogramService

PersistencyService

DataFiles

TransientHistogram

Store

ApplicationManager

ConverterConverterEventSelector



- Main applications have similar scope: event generation, detector simulation, reconstruction, analysis- Both experiments use Configuration Management Tool (CMT) for code organisation, but with differences in how it’s used- Experiments have similar views on job building, but have different requirements for implementation: ATLAS favours use of Analysis Job Description Language (AJDL), a key ingredient of ATLAS Distributed Analysis (ADA) system LHCb happy with more-direct mapping to job scripts


Job building: LHCb

DaVinciWorkflow

SelectWorkFlow

PrepareSandbox

PrepareAlgFlowOptions

and DLLs

DaVinciAlgorithmFlow

EditAlgorithmFlow

AlgParamOptions

SelectDatasets

EditAlgParamOptions

DatasetOptions

AlgFlowOptions

SandboxDLLs

JobOptionsFileCatalog slice

Submit Job

Metadatacatalog

AlgOptionscatalog

DLLs

Filecatalog


Job building: ATLAS


AJDL in outline

- AJDL (Analysis Job Description Language) provides for high-level description of job content- Basic elements include: Application: name and version number Task: user configuration of application Dataset: data to be processed Preferences: resource requirements- AJDL aims to define way in which job information is transmitted between software components:

Allow multiple clients to communicate with independently implemented services by defining a common interface- AJDL represents job data using XML

DIAL provides C++ classes that interpret and manipulate the XML


Client-service communication via AJDL

PI/SEAL

GANGA

ROOT

JAS

Command line

AJDL

PROOFARDA

Condor-G

GCE/ChimeraSTARJDAP (JAS)

DIAL-interactive

Analysis environments Analysis services

Webservices

ATLAS production

Plug-inclients

Portal/switch

Initial ADA



- Experiments have different metadata catalogues ATLAS has AMI LHCb has bookkeeping database Different client tools for extracting information- Both experiments see Ganga as front-end to distributed analysis system: ATLAS interested in Ganga as front-end to web services with AJDL interface LHCb interested in Ganga as front-end to DIRAC Workload Management System


Ganga as front-end for ATLAS system

Middleware service interfaces

CEWMS FileCatalog

etc. ...etc. Middlewareservices

High level service interfaces (AJDL)

DIALAnalysisService

GANGAAnalysisService

ATPRODAnalysisService

ROOTcmd line

Client

GANGAcmd line

Client

GANGATask

Management

CaponeAnalysisService

GANGAJob

Submission

GANGAJob

Management

High-levelservices

Client tools

ARDAAnalysisServiceCatalog

services

GANGA GUI

DatasetSplitter

DatasetMerger

JobManagement

From GANGA


Ganga as front-end for LHCb system

DIRAC JobManagement

Service

DIRAC JobManagement

Service

DIRAC CEDIRAC CEDIRAC CEDIRAC CE

DIRAC CEDIRAC CE

LCGLCGResourceBroker

ResourceBroker

CE 1CE 1

DIRAC SitesDIRAC Sites

AgentAgent AgentAgent AgentAgent

CE 2CE 2

CE 3CE 3

Productionmanager

Productionmanager GANGA UIGANGA UI User CLI User CLI

JobMonitorSvcJobMonitorSvc

JobAccountingSvcJobAccountingSvc

AccountingDB

Job monitorJob monitor

InfomarionSvcInfomarionSvc

FileCatalogSvcFileCatalogSvc

MonitoringSvcMonitoringSvc

BookkeepingSvcBookkeepingSvc

BK query webpage BK query webpage

FileCatalogbrowser

FileCatalogbrowser

Userinterfaces

DIRACservices

DIRACresources

DIRAC StorageDIRAC Storage

DiskFileDiskFile

gridftpgridftpbbftpbbftp

rfiorfio



- Similarities between ATLAS and LHCb are motivation for common project- Differences have led to a component-based design, capable of satisfying a wide range of requirements Core components of general applicability supplemented by experiment-specific components Approach works well for ATLAS and LHCb Ganga has also been used successfully in BaBar (J.Martyniak)


Ganga design: schematic

Job Definition

Job Registry

Job Handling

File Transfer

Python Native

Softw

are Bus

CLI

GUI

Py

RO

OT

Gau

di

Pyth

on

PyC

MT

PyA

MI

PyD

ial

BaBar Job Definition and

Splitting

Gaudi/Athena Job Options

Editor

Gaudi/Athena Job Definition


Ganga core classes

JobHandler

Requirements

JobAttributes

Credentials

1

1…

Job

JobsRegistry

Application

ApplicationHandler

Parameter

Executable1

0…

1

0…

1

1

1

1…

Job definition component

Job registry component

Job handling component

File

FileHandler

0…

1

Configuration

Options

ExtraOptions

1

1

Application handling component

0, 1

0, 1

1


Example Ganga handler classes

JobHandler

GridJobHandler

PBSJobHandler

LSFJobHandler

Job handling component

ApplicationHandler

BaBarApplicationHandler

GaudiApplicationHandler

DaVinchiApplicationHandler

AtlfastApplicationHandler

Application specific components

BatchJobHandler

LocalJobHandler

WMSJobHandler

EDGJobHandler


Release history

- Ganga 1.0 released in July 2003 Basic functionality implemented Tutorial given at US-ATLAS Computing and Physics Meeting, BNL, 27th August 2003, with about 50 participants Tutorial provided useful feedback to developers- Major overhaul and upgrade begun in September 2003- Ganga 2.0 released May 2004, and followed by several minor releases Improved GUI, revised core classes, better stability, additional functionality Ganga 2.3 released 13th September 2004- Ganga 3.0 scheduled for October 2004 Interfaces to experiment tools and services, user-friendly command-line interface


Ganga 2.x

- Ganga 2.x is a working system: download and try!- Installation instructions and user guides available from Ganga web site: http://ganga.web.cern.ch/ganga/user/v2/QuickStart.html- Main features include: Graphical User Interface (GUI) Command echoing Job-options editor (JOE) Job submission to different batch systems (localhost, LSF, PBS) Job submission to Grid via DIRAC, in experimental form Example of aided application configuration, for LHCb analysis Interface to LHCb metadata catalogue Job catalogue Automatic monitoring

http://ganga.web.cern.ch/ganga/user/v2/QuickStart.html


Job creation


Job definition


Examination of job output


Interface to LHCb metadata catalogue


Job-options editor: main features

- Standalone Qt-based graphical editor, available from Ganga- Presents hierarchical view of available options, and helps user with value entry- Multiple job-options files may be edited simultaneously- Support for both text and Python job-options files- Files saved with included files expanded, so that result is portable


JOE: main window


JOE: option entry


Work in progress

- Work in progress in several areas: AS: Job submission to DIRAC KH, CLT: Job submission via AJDL interface JM: User-friendly command-line interface CC-T, CLT: Interface to ATLAS metadata catalogue (AMI) KC: Output merger GR: Package-management tool WL: Upgrade of component manager (PyBus)- Last two aren’t strictly Ganga activities, but will benefit Ganga


Job submission to DIRAC

- Job handler for submission to DIRAC is already available in Ganga 2.x, but testing is still in progress- Handler creates job-description files required by DIRAC- Job submission and monitoring performed making direct use of DIRAC API- Output files retrieved automatically when ready, in separate thread- Log file is checked for error messages, then job state is updated


Job submission via AJDL interface

- Python client for AJDL services developed by producing bindings for DIAL C++ classes Generate LCG dictionaries using lcgdict Compile resulting _dict.cpp files against ReflectionBuilder to create libraries that can be imported into Python using PyLCGDict2 Procedure now largely automated- Have added set of Python functions to simplify the use of some of the classes For example, can submit job to remove DIAL server (GSI authentication) and query status with: import pydial pydial.submit(“dataset.xml”,”task.xml”,”application.xml”) pydial.showStatus()


Prototype GUI for AJDL job builder

Working on appropriate handlers for connection to Ganga


Interface to ATLAS metadata catalogue

- Python client for ATLAS metadata catalogue (AMI) developed- Allows use of main AMI methods for querying and updating catalogue- Checks for mandatory parameters are made before a command is sent to AMI- XML results returned by AMI are automatically converted to appropriate Python object


Prototype GUI for Ganga interface to AMI

- Currently very simple- Allows user to select AMI project / processing step- User builds query- User selects LFNs and

Ganga adds them to a job script


Output merger

- Output merger is being developed as a standalone application, Josie (more a companion for JOE than an acronym), which will be available from Ganga- Core classes have been implemented- Currently, can produce macros for merging ROOT ntuples and histograms, concatenate text files, and perform text filtering- Functionality available via command line and GUI


User-friendly command-line interface

- Ganga echoes commands executed when operations are performed through the GUI- Echoed commands can be entered directly at Python prompt, but can be cumbersome- Draft scheme for more-intuitive command-line interface has been written, and a prototype implementation is available


Conclusions

- Differences between ATLAS and LHCb create challenges for Ganga, but these are being met- Significant Ganga-related activity in both experiments GridPP developers have key roles, but also others are making

major contributions- Ganga release 2.x is a working system Installation instructions and user guide available from Ganga web

site- Ganga 3.0, with submission to DIRAC and via AJDL interface, scheduled for October- LHCb-UK software course to be held at Cambridge in December Ganga will be used as tool for submitting jobs

K. Harrison Liverpool, 14th September 2004 GANGA DEVELOPMENT - Objectives and organisation - Ganga in ATLAS and LHCb - Release status - Work in progress.

Documents

atlas slide

job data

job options lhcb

job building

job information

text job options atlas

atlas favours

adams atlas