DataTag is a project funded by the DataTag is a project funded by the European Union European Union CHEP 2003 – 24-28 March CHEP 2003 – 24-28 March 2003 - 2003 - The WorldGRID transatlantic testbed A successful example of Grid interoperability across EU and US domains Flavia Donno (Formerly of DataTAG WP4, LCG) [email protected]http://chep03.ucsd.edu/files/249.ppt CHEP 2003 – 24-28 March – n CHEP 2003 – 24-28 March – n o . (1) . (1)
34
Embed
The WorldGRID transatlantic testbed A successful example of Grid interoperability
The WorldGRID transatlantic testbed A successful example of Grid interoperability across EU and US domains . Flavia Donno (Formerly of DataTAG WP4, LCG) [email protected] http://chep03.ucsd.edu/files/249.ppt. CHEP 2003 – 24-28 March – n o . (1). Motivation Participants - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
DataTag is a project funded by the European UnionDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 -CHEP 2003 – 24-28 March 2003 -
The WorldGRID transatlantic testbedA successful example of Grid interoperabilityacross EU and US domains
CHEP 2003 – 24-28 March – nCHEP 2003 – 24-28 March – noo. (1). (1)
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 22
Talk Outline
• Motivation
• Participants
• Interoperability issues
• Solutions
• Architecture
• Monitoring/Support
• Spin off
F. DonnoCERN/IT and INFN
R. GardnerUniversity of Chicago
Applications
CMS
ATLAS
Monitoring with Nagios
Monitoring with Ganglia
Conclusions
Next Steps
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 33
ParticipantsParticipants
DataTAG:F. Donno, CERN - INFNS. Andreozzi, R. Barbera, V. Ciaschini, S. Fantinel, A. Ghiselli, M. Mazzucato, D. Rebatto, G. Tortone, L. Vaccarossa, M. Verlato, C. Vistoli – INFNM. Draoli, CNR-Rome
CrossGrid:A. Garcia, M. Hardt, FZK - GermanyJ. Marco, UC - Spain M.David, J. Gomes, LIP - PortugalO. Maroney, U.Bristol, UK
Trillium/iVDGL:P. Avery, J. Rodriguez - U. FloridaE. Deelman, N. Olomu - USC/ISIJ. Gieraltowski, S. Gose, E. May, J. Schopf – ArgonneAfaq, J. Annis, R. Glossum, R. Pordes, V. Sekrhi – FermilabW. Deng, J. Smith, D. Yu - BNL A. DeSmit, A. Roy - Wisconsin C. Dumitrescu, I. Foster, R. Gardner, U. ChicagoL. Grundhoefer ,J. Hicks, F. Luehring, L. Meehan - U. IndianaS. Youssef, Boston UniversityB. Moe - MilwaukeeD. Olson – LBNLS. Singh - Caltech
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 44
MotivationsMotivations
Basic collaboration between European and US Grid projects
Interoperability between Grid domains for applications submitted by users from different virtual organizations Controlled use of shared resources subject to agreed policy Integrated use of heterogeneous resources from iVDGL and DataGrid/CrossGrid testbed domains
Goal:Goal:
Build a “transatlantic grid” based on the existent European Build a “transatlantic grid” based on the existent European and American Grids with the goal of offering transparent and American Grids with the goal of offering transparent access to the distributed computing infrastructure necessary access to the distributed computing infrastructure necessary to the “data-intensive” modern applicationsto the “data-intensive” modern applications
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 55
Interoperability IssuesInteroperability Issues Many grids with several OSseveral OS (RH 6.2, RH 7.x, Fermi Linux, CERN Linux,…), several compilers and software components.Different Grid ArchitecturesGrid Architectures (VDT server/client vs. Computing Elements, Storage Elements, User Interfaces, …)Need to identify minimum set of core servicesminimum set of core services and define collective/optional servicesoptional services Common protocols/Same or compatible versions of the softwareAuthentication and AuthorizationAuthentication and Authorization mechanism: authority trusting, user authentication/authorization via LDAP VO Servers.Grid resource descriptionGrid resource description/status: Globus schema vs. EDG schema vs. GLUE schema Several Grid Data managementGrid Data management ToolsSoftware distribution and configurationSoftware distribution and configuration : rpm based vs. PACMAN
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 66
SolutionsSolutions
Many grids with several OSseveral OS (RH 6.2, RH 7.x, Fermi Linux, CERN Linux,…), several compilers and software components.
Partition WorldGrid in subdomains with uniform or compatible set of basic services. Such resources will advertise themselves with specific targets to the applications (such as RH6.2).Try to keep the subdomains as large as possible.
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 77
SolutionsSolutions
Different Grid ArchitecturesGrid Architectures (VDT server/client vs. Computing Elements, Storage Elements, User Interfaces, …)
CECE
UIUI
SESERBRB
VDT Client
VDT Server
RCRC
ISIS
RCRC
ISIS
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 88
SolutionsSolutions
Need to identify minimum set of core servicesminimum set of core services and define collective/optional servicesoptional services Common protocols/Same or compatible versions of the software
Globus and Condor core services (GRAM, GSI, MDS, GridFTP, …)
Resource Broker, User Interface and JDL, Data Management high level tools (edg-replica-manager, MAGDA, Globus Replica Catalog, …) collective optional services not installed universally
User Grid Portals (Genius, GRAPPA, …): a variety available not to change the User Interface to the GRID
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 99
SolutionsSolutions
Authentication and AuthorizationAuthentication and Authorization mechanism: authority trusting, user authentication/authorization via LDAP VO Servers.
DOE and EDG certificates universally accepted DataTAG and iVDGL VO LDAP servers trusted mkgridmap tool universally installed Local security policy sites agreed to allow
access to grid demonstration users (kerberos, …)
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 1010
SolutionsSolutions
Grid resource descriptionGrid resource description/status: Globus schema vs. EDG schema vs. GLUE schema
three coexistent schemas in place (Globus, EDG, GLUE) installed on all resources
Some tool (monitoring) working with all of them
EDG middleware using both EDG and GLUE US tools using none or Globus
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 1111
SolutionsSolutions
Software distribution and configurationSoftware distribution and configuration : rpm based vs. PACMAN
Created WorldGrid distribution (rpm/LCFGng and PACMAN)
Effort to ensure coherency and automatic configuration
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 1212
Final ArchitectureFinal Architecture
CECE
UIUI
SESERBRB
VDT Client
VDT Server
RCRC
ISIS
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 1313
Monitoring and SupportMonitoring and Support
Two monitoring tools VO based in place: Two monitoring tools VO based in place: edt-monitor based edt-monitor based on Nagios and iVDGL based on Ganglia (see talk from R. on Nagios and iVDGL based on Ganglia (see talk from R. Gardner)Gardner)
Support infrastructure: Support infrastructure: to support to support
site site administrators administrators during the during the installation and installation and configuration configuration procedure. Also procedure. Also for problem for problem fixing fixing during during normal normal operationoperation
DataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – nDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 – noo 1414
Spin-offSpin-off GLUE schema:GLUE schema: WorldGrid has allowed to prove the validity of WorldGrid has allowed to prove the validity of the GLUE schema and encouraged EDG to deploy itthe GLUE schema and encouraged EDG to deploy it VOMS:VOMS: The authentication/authorization problems were The authentication/authorization problems were identified and parallel research activities started, like the one on identified and parallel research activities started, like the one on Virtual Organization Manager ServiceVirtual Organization Manager Service GLUE Packaging:GLUE Packaging: A working group is trying to find a solution A working group is trying to find a solution for a standardization of the packaging, distribution and for a standardization of the packaging, distribution and configuration problem for a software releaseconfiguration problem for a software release GLUE Testing:GLUE Testing: The problem of verifying an installation and The problem of verifying an installation and validate a site for joining the Grid has been addressed and a validate a site for joining the Grid has been addressed and a working group has startedworking group has started Support:Support: A first operation/monitoring center has started A first operation/monitoring center has started in US taking advantage of the in US taking advantage of the monitoring toolsmonitoring tools. Other . Other centers in EUcenters in EU LCG-0:LCG-0: After the demonstration at IST2002 and SC2002, After the demonstration at IST2002 and SC2002, LCG has based his LCG has based his first middleware distribution onfirst middleware distribution on the the WorldGridWorldGrid experience experience
DataTag is a project funded by the European UnionDataTag is a project funded by the European Union CHEP 2003 – 24-28 March 2003 -CHEP 2003 – 24-28 March 2003 -
Rob Gardner University of Chicago
on behalf of the WG group
The WorldGRID transatlantic testbed, Part 2A successful example of Grid interoperabilityacross EU and US domains
16 R. Gardner CHEP 2003 – 24-28 March 2003 – nR. Gardner CHEP 2003 – 24-28 March 2003 – noo 1616
Talk Outline
Motivation
Participants
Interoperability issues
Solutions
Architecture
Monitoring/Support
Spin off
F. DonnoCERN/IT and INFN
Part 2
R. GardnerUniversity of Chicago
Applications
CMS
ATLAS
Monitoring with Nagios
Monitoring with Ganglia
Conclusions
Next Steps
17 R. Gardner CHEP 2003 – 24-28 March 2003 – nR. Gardner CHEP 2003 – 24-28 March 2003 – noo 1717
Installing Apps on 2 Grids We needed a way to get applications from three
experiments (VO’s) setup on the execution sites On DataTAG resources, selected CE’s were loaded
with CMS or ATLAS rpms On iVDGL resources, we Pacmanized binaries
(rpms and tarballs) of bundled applications– %pacman –get iVDGL:ScienceGrid
> Atlas-kit, Atlas-ATLFAST
> CMS-MOP, EDG-CMS
> SDSS Astrotools
– binaries, and run time environments
3 experiments
18 R. Gardner CHEP 2003 – 24-28 March 2003 – nR. Gardner CHEP 2003 – 24-28 March 2003 – noo 1818
ATLSIMJob
Write to Grid
Storage Element
Read from Grid
Storage Element Input Data
Output ZEBRA
Grid Storage
Apache
EnginFrame
GENIUS
https+java/xml+rfb
WEB Browser
EDGUI
Local
WS
the GridEDG+GSI
ATLAS and CMS with GENIUS
see R. Barbera’s Genius talk this conference
19 R. Gardner CHEP 2003 – 24-28 March 2003 – nR. Gardner CHEP 2003 – 24-28 March 2003 – noo 1919