1 New developments in the HENP-GC HENP-GC Collaboration New Capabilities in the New Capabilities in the HENP Grand Challenge Storage Access HENP Grand Challenge Storage Access System System and its Application at RHIC and its Application at RHIC David Malon, Argonne presenting for the HENP-GC collaboration* (http:/www- rnc . lbl . gov /GC/ ) 8 Feb 2000 CHEP *slides thanks to L. Bernardo, D. Olson, A. Shoshani, S. Vanyashin
28
Embed
New Capabilities in the HENP Grand Challenge Storage Access System and its Application at RHIC
New Capabilities in the HENP Grand Challenge Storage Access System and its Application at RHIC. David Malon, Argonne presenting for the HENP-GC collaboration* ( http:/www-rnc.lbl.gov/GC/ ) 8 Feb 2000 CHEP *slides thanks to L. Bernardo, D. Olson, A. Shoshani, S. Vanyashin. Outline. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1New developments in the HENP-GC HENP-GC Collaboration
New Capabilities in the New Capabilities in the HENP Grand Challenge Storage Access System HENP Grand Challenge Storage Access System
and its Application at RHICand its Application at RHIC
David Malon, Argonne presenting for the HENP-GC collaboration*
(http:/www-rnc.lbl.gov/GC/)
8 Feb 2000
CHEP*slides thanks to L. Bernardo, D. Olson, A. Shoshani, S. Vanyashin
2New developments in the HENP-GC HENP-GC Collaboration
OutlineOutline
• Overview of HENP-GC
• What’s new since CHEP’98
• STACS
• Experiment interface
• Scalability testing
• Conclusion
3New developments in the HENP-GC HENP-GC Collaboration
High-Energy & Nuclear Physics Grand ChallengeHigh-Energy & Nuclear Physics Grand Challenge
• 3 year project, mid-1997 to mid-2000
• Funded by DOE/MICS with contributed effort from DOE/HENP
• Participants:– NERSC/Berkeley Lab
• L. Bernardo, A. Mueller, H. Nordberg, A. Shoshani, A. Sim, J. Wu
– Argonne• D. Malon, E. May, G. Pandola
– Brookhaven Lab• B. Gibbard, S. Johnson, J. Porter, T. Wenaus
– Nuclear Science/Berkeley Lab• D. Olson, A. Vaniachine, J. Yang, D. Zimmerman
4New developments in the HENP-GC HENP-GC Collaboration
What is the Grand Challenge architecture?What is the Grand Challenge architecture?
• An order-optimized prefetch architecture for data retrieval from multilevel storage in a multiuser environment
• Queries select events and specific event components based upon tag attribute ranges– query estimates are provided prior to execution– collections as queries are also supported
• Because event components are distributed over several files, processing an event requires delivery of a “bundle” of files
• Events are delivered in an order that takes advantage of what is already on disk, and multiuser policy-based prefetching of further data from tertiary storage
• GCA intercomponent communication is CORBA-based, but physicists are shielded from this layer
5New developments in the HENP-GC HENP-GC Collaboration
System OverviewSystem Overview
Client
GCASTACS
Stagedeventfiles
EventTags (Other)
disk-residentevent data
Index
HPSSpftp
File Catalog
ClientClient
Client
Client
6New developments in the HENP-GC HENP-GC Collaboration
STorage Access Coordination System STorage Access Coordination System (STACS)(STACS)
QueryEstimator
QueryMonitor
CacheManager
PolicyModule
Bit-SlicedIndex
FileCatalog
QueryStatus,CacheMap
Query
Estimate
File Bundles,Event lists
Pftp andfile purgecommands
List offile bundles and events
Requests forfile cachingand purging
7New developments in the HENP-GC HENP-GC Collaboration
What is new since CHEP’98What is new since CHEP’98
• Multi-component multi-file event model– Event is composed of separate components– Components of a single event are stored in separate
files
• Removal of Objectivity/DB-specific dependencies– eventID is an experiment-specific typedef– CORBA file catalog interface
• User-accessible file bundle information for user-code- dependent file I/O
– Order-Optimized Iterator: delivery of event ids (and optional file information) as bundles are cached
20New developments in the HENP-GC HENP-GC Collaboration
Iterator ExtensionsIterator Extensions
• // iterator is initialized with this query's token, and a // pointer to GCA_Resources for access to remote STACS //components and configuration parameters:
while (GCIter.next(eventID& myEvent){ usercode(myEvent); // process an event
}
• In order to allow for experiment-specific code to handle file I/O an optional flag that signals a new file bundle is used along with a method to retrieve the file name for each event component:
21New developments in the HENP-GC HENP-GC Collaboration
STAR uses the fileCatalog & instance tables in STAR uses the fileCatalog & instance tables in MySQL to satisfy the fcFileCatalog CORBA interfaceMySQL to satisfy the fcFileCatalog CORBA interface
// fileCat.idl// Luis Bernardo <[email protected]>, Alex Sim <[email protected]>// Lawrence Berkeley National Laboratory// May 99// Purpose: defines interface between File Catalog (server) and Cache// Manager, Query Estimator and Query Monitor (clients).