CHEP 2004, Core Software Integration of POOL into Integration of POOL into three Experiment Software three Experiment Software Frameworks Frameworks Giacomo Govi CERN IT-DB & LCG-POOL K. Karr, D. Malon, A. Vaniachine (Argonne National Laboratory) P. Van Gemmeren (BNL) R. Chytracek, D. Duellmann, M. Frank, M. Girone, G. Govi, V. Innocente, P. Mato Vila, J. Moscicki, I. Papadopoulos, H. Schmuecker (CERN) R D Schaffer (LAL-IN2P3) Z. Xie (Princeton University ) T. Barrass (University of Bristol) C. Cioffi (University of Oxford) W. Tanenbaum (Fermi National Accelerator Laboratory)
17
Embed
CHEP 2004, Core Software Integration of POOL into three Experiment Software Frameworks Giacomo Govi CERN IT-DB & LCG-POOL K. Karr, D. Malon, A. Vaniachine.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CHEP 2004, Core Software
Integration of POOL into three Integration of POOL into three Experiment Software FrameworksExperiment Software Frameworks
Giacomo GoviCERN IT-DB & LCG-POOL
K. Karr, D. Malon, A. Vaniachine (Argonne National Laboratory)P. Van Gemmeren (BNL)R. Chytracek, D. Duellmann, M. Frank, M. Girone, G. Govi, V. Innocente, P. Mato Vila, J. Moscicki, I. Papadopoulos, H. Schmuecker (CERN)R D Schaffer (LAL-IN2P3) Z. Xie (Princeton University )T. Barrass (University of Bristol)C. Cioffi (University of Oxford)W. Tanenbaum (Fermi National Accelerator Laboratory)
• Provide a framework for C++ object persistency - - API neutral on storage technology - Root and RDBMS backends
- File access integrated with current GRID technologies
• Follow up experiment specific requirements - Extract ‘synthesis’ among many (overlapping) use cases
- Resolve conflicting requirements
- Find common solutions
• Encourage concrete experiment participation - - Include experiment members in the POOL Core developers team - Follow up quick integration of POOL releases in the experiment framework - Involve experiment in the validation phase of release processes. • Re-use experience from previously adopted Persistency-
ATLAS: The Overall approach of the experiment framework on object Persistency
From the Athena/Gaudi framework point of view, POOL is just a new I/O “technology” This implies writing a new conversion service Main components:
AthenaPoolCnvSvc - conversion service AthenaPoolConverter - converter base class T_AthenaPoolCnv<T> - templated converters PoolSvc - Athena/Gaudi service interface to POOL
Allows jobOptions DataHeader - stores the refs of the Event Data Objects
Ref to DataHeader is inserted in the event collection
ATLAS has simplified the user interface by allowing “generic” converters: Use templated converter and generate the necessary classes to create the
converter automatically User just needs to specify a “.h” file for each DataObject (pool ref’ed object)
POOL DataSvcPOOL DataSvc is for now the entry point for object persistency is for now the entry point for object persistency Two caches: input/output, use Ref<T> for object I/O Cache functionality is not strictly needed => duplicates Athena/Gaudi transient store
Object lifetime managed by Athena/Gaudi transient store For event storage, using: RootStorageSvc and Implicit collections Some conditions storage using RootStorageSvc
Support both Tree-based and Key-based ROOT storage, selectable in Athena via JobOptions Using ROOT Trees as default for now to gain experience Key-based approach is similar to what has already been tested in ATLAS via Objectivity Expect to also use Object/Relational storage when available
Using XML catalogs for local data access, EDG RLS and Globus RLS for master file catalogs
For tag collections: using both Root and MySQL collections Currently deploying Oracle DB for detector description parameters via Relational
Access Layer
G.G
ovi,
IT-D
B
Slide 7
Use of POOL in CMS: current statusUse of POOL in CMS: current status
COBRA OSCAR ORCA Current version being tested using last POOL 2.0.0 internal release
Production still use pool 1.7
Usable for production, deployed to physicists Used for SW tutorials each Friday since autumn 2003 35 Million events produced with OSCAR (G4 simulation)
Essentially same functionality as previous Objectivity-based code Limitations
No concurrent update of databases▫ No direct connection to central database while running
Remote access limited to RFIO or dCache (soon GFAL?) No Schema evolution
Added values No need of a common base class (ooObj) Native support of stl containers Support for transient attributes
G.G
ovi,
IT-D
B
Slide 8
Algorithm Context
(Event)
pool Refs
Local transient store
Persistent store
POOL
DataSvc
(Object cache)
Reconstruction on demand
Data Access in CMSData Access in CMS
Retrieve a chunk
G.G
ovi,
IT-D
B
Slide 9
What CMS uses of POOL What CMS uses of POOL
Transition from Objectivity to Pool inspired by a minimum impact principle
• All objects (event and metadata) are stores as root keyed-objects (no root-tree)
• Only object navigation is used, no other access mechanisms• Ref
Full interface• File Catalog
Full interface XML implementation in Physics Applications MySQL & RLS used in production (DC04)
• Session Only Transaction Management No explicit Database/Container handling
CHEP 2004, Core Software G.Govi, IT-DB 10
LHCb Goals for POOL Integration
• Keep the existing framework architecture– Objects are transient and reside in “Data Stores”
• source for conversion to persistent or graphical representation– “Algorithms” access objects by “logical name”
from a data store
• Keep the existing event model description– Code (headers) generated from XML files
• Need to access existing data– Read data with pre-POOL software (ROOT based)
• Usage of POOL transparent to end-users (physicists)
CHEP 2004, Core Software G.Govi, IT-DB 11
Data Access in Gaudi Applications
POOL
(5) Register
DataService
Algorithm
(1) retrieveObject(…)
Try to accessan object data
(2) Search in Store
Data Store
(3) Request load
PersistencyService
ConversionService
Technology dispatcher
PersistencyService
PersistencyService
StorageServiceStorageService
ConversionService(s)
ConversionService(s)
CHEP 2004, Core Software G.Govi, IT-DB 12
Customization of POOL for Gaudi• Currently main technology for event storage
– Write event data in ROOT tree-mode– Detector data etc. not (yet) implemented
Learning from Integration ExperiencesLearning from Integration Experiences
Differences are large in the object transient store and navigation area. -The navigation mechanism strongly influences the Cache features (object lifetime management, semantics for object association)
-Some experiment framework already had specific object bookkeeping services – not easily re-usable for common purposes
-A real decoupling of the Ref implementation from the Cache details is difficult to achieve
-A review of the top level architecture and API (done in 2003 with the experiments) could not find a common solution