OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
Post on 04-Apr-2018
220 Views
Preview:
Transcript
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
1/15
2006 Open Grid Forum
OGF-Europe Tutorial: How new
communities can get access to a GridinfrastructureNorduGrid - NDGF
Oxana Smirnova
NDGF / Lund UniversityOGF25
March 2 2009, Catania, Italy
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
2/15
2006 Open Grid Forum
OGF IPR Policies Apply
I acknowledge that participation in this meeting is subject to the OGF Intellectual Property Policy. Intellectual Property Notices Note Well: All statements related to the activities of the OGF and
addressed to the OGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the OGFand its participants certain licenses and rights in such statements. Such statements include verbalstatements in OGF meetings, as well as written and electronic communications made at any time orplace, which are addressed to:
the OGF plenary session, any OGF working group or portion thereof, the OGF Board of Directors, the GFSG, or any member thereof on behalf of the OGF, the ADCOM, or any member thereof on behalf of the ADCOM, any OGF mailing list, including any group list, or any other list functioning under OGF auspices, the OGF Editor or the document authoring and review process
Statements made outside of a OGF meeting, mailing list or other function, that are clearly not intendedto be input to an OGF activity, group or function, are not subject to these provisions. Excerpt from Appendix B of GFD-C.1: Where the OGF knows of rights, or claimed rights, the OGF
secretariat shall attempt to obtain from the claimant of such rights, a written assurance that uponapproval by the GFSG of the relevant OGF document(s), any party will be able to obtain the right toimplement, use and distribute the technology or works when implementing, using or distributingtechnology based upon the specific specification(s) under openly specified, reasonable, non-discriminatory terms. The working group or research group proposing the use of the technology withrespect to which the proprietary rights are claimed may assist the OGF secretariat in this effort. Theresults of this procedure shall not affect advancement of document, except that the GFSG may deferapproval where a delay may facilitate the obtaining of such assurances. The results will, however, be
recorded by the OGF Secretariat, and made available. The GFSG may also direct that a summary of theresults be included in any GFD published containing the specification.
OGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that supportthe Internet Standards Process.
2009-03-02 2www.ndgf.org www.nordugrid.org
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
3/15
2006 Open Grid Forum
Outlook
Basic production middleware overview:
Advanced Resource Connector (ARC)
NDGF and its users:the WLCG Tier1 and more
On the road towards standards-based
interoperability
www.ndgf.org www.nordugrid.org2009-03-02 3
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
4/15
2006 Open Grid Forum
NorduGrid, NDGF and others
NorduGrid an academic researchcollaboration that started in 2001 andspawned a number of projects
Develops and maintains the AdvancedResource Connector (ARC) Gridmiddleware
NDGF Nordic Grid Infrastructure
funded through the Nordic Council ofMinisters Provides Tier1 for LHC computing,
coordinates Tier2s
Makes use of ARC as an underlyingtechnology
EU KnowARC R&D project
providing new standard interfaces toARC
NGIn project funded by Nordunet3;focuses on ARC education andapplications
www.ndgf.org www.nordugrid.org2009-03-02 4
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
5/15
2006 Open Grid Forum
How to build a Grid:
ARC in a nutshell
2009-03-02 5www.ndgf.org www.nordugrid.org
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
6/15
2006 Open Grid Forum
Production ARC overview
Current ARC version in production: v0.6.5
Available freely at http://www.nordugrid.org
Open Source (GPL v2, next versions - Apache 2.0)
Binary packages for20+ Linux flavors/versions
(RH, Fedora, SuSE, Debian, Ubuntu)
Reliable implementation of fundamental Grid functionalities:
De-factostandard Grid security: GSI, VOMS, GACL (needs 3rd
party libraries) Job submission: by matchmaking/brokering or direct
Job monitoring, logging and life cycle management
Information services: resource aggregation, representation, discovery and
monitoring
Basic data management:
Automated seamless input/output data movement Interface to data indexing services (e.g. LFC), client-side data movement
Storage Elements (GridFTP, SRM own or 3rd party) NDGF uses dCache
Builds upon standard Open Source solutions and protocols:
Globus Toolkit pre-WS API and libraries (not services! )
OpenLDAP, OpenSSL, SASL, SOAP, GridFTP, GSI
2009-03-02 6www.ndgf.org www.nordugrid.org
http://www.nordugrid.org/http://www.nordugrid.org/7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
7/15
2006 Open Grid Forum
How new communities join
the Grid
No centralizedcontrol whatsoever No central
operations No databases of
resources or VOs
No availability orreliability tests
Typically, userscome via nationalGrid initiatives: NDGF, SweGrid,
SwiNG (Swiss),
UkrainianAcademic Grid etc
2009-03-02 7www.ndgf.org www.nordugrid.org
Educate users
Organize tutorials, workshops involving middleware experts
Deploy middleware over your computing facilities
Get additional hardware, ifneeded
Get additional system experts, ifneeded
If necessary, publish sites in theNorduGrid information system
Identify application projects
Can be on a competitive basis, via calls for proposals
Identify your Certification Authority
Must be IGTF-endorsed For tests, an Instant CA will do
Identify your resources
Computing facilities Storage facilities Manpower
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
8/15
2006 Open Grid Forum
Nordic DataGrid Facility
Provides a Grid infrastructure, incl.a unique distributed Tier1 centre
Involves 7 largest Nordic academicHPC centres
plus a handful of Universitycentres (Tier2 service)
Inter-Nordic shared 10Gbit networkfrom NORDUnet
Budget: staff only, 2 MEUR/year, byNordic research councils
www.ndgf.org www.nordugrid.org2009-03-02 8
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
9/15
2006 Open Grid Forum
NDGF Tier1 services overview
Level 1: central services,99% up-time
dCache: admin,doors: gridftp xrootd,srm
Data indexing: LFC
WLCG monitoring:site BDII
Level 2: distributed,98% up-time
Storage pools
ARC gatekeepers
VOBoxes (ALICE)
FTS: with GridFTPv2
WLCG 3D database
VOMS
Level 3: distributed,95% up-time
Worker nodes
ARC GIISes
Monitors
www.ndgf.org www.nordugrid.org2009-03-02 9
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
10/15
2006 Open Grid Forum
Example of a country
contribution: SweGrid
Investment Date Cost,KEUR
Six clusters (6x100 cores)including 12 TB FC disk Dec 2003 1 120Disk storage part 1, 60 TB SATA May 2004 320Disk storage part 2, 86.4 TB SATA May 2005 230
Centre Tapevolume, TB
Cost,
KEURHPC2N 120 100PDC 120 100NSC 120 100
SweGrid in 2003-2007Location ProfileHPC2N (Ume) IT
UPPMAX (Uppsala) IT, HEP
PDC (Stockholm) IT
C3SE (Gothenburg) IT
NSC (Linkping) IT
Lunarc (Lund) IT, HEP
Co-funded by the Swedish
Research Council and the Knutand Alice Wallenberg foundation One technician per centerMiddleware: ARC, dCache, gLite 1/3 allocated to LHC Computing
www.ndgf.org www.nordugrid.org2009-03-02 10
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
11/15
2006 Open Grid Forum
User communities
LHC ChemistryOther Physics Amanda/IcecubeGeo Science Fluid MechanicsBioinformatics MedicineComputer Science BiotechnologyMathematics StatisticsPharmacology Material Chemistry
Electronics
www.ndgf.org www.nordugrid.org2009-03-02 11
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
12/15
2006 Open Grid Forum
Enabling applications
Application-specific software is formalized as Run-TimeEnvironment (RTE)
Installed and configured at each site, typically by anauthorized user/manager
Advertised via information system for match-making
Jobs only need to specify the RTE name and version
Special authorization plugins are available in case of a
commercial application software Users with similar applications are grouped into Virtual
Organisations (VOs) A VO typically corresponds to a project
A VO contributes the hardware
Some VOs at NDGF: CERN VOs: ALICE, ATLAS, CMS
Gene databases for bio-informatics sciences Screening of CO2-Sequestration suitable reservoirs
Computational Chemistry
2009-03-02 12www.ndgf.org www.nordugrid.org
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
13/15
2006 Open Grid Forum
On interoperability
Strategy: interoperability via open standards BES, JSDL, GLUE2, SRM, GridFTP, X509, SAML etc Aggreed extensions are critical (profiles) Tip: attend the Production Grid Infrastructure (PGI-WG)
workshop on Wednesday Shorter term: transitional gateway-like solutions are
available (ARC-gLite) Currently in development:
ARC client library addresses the ARC other middlewaredirection
CLI will offer transparent access capability to 3rd partyservices
Primary target platforms: gLite, Unicore New ARC client can already now submit jobs to CREAM,
Unicore compute elements
2009-03-02 13www.ndgf.org www.nordugrid.org
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
14/15
2006 Open Grid Forum
Future perspectives
ARC evolves from a pre-WS solution to a WebService based one PGI-WG work ensures that it will be standards-
conformant and interoperable with other standards-conformant middlewares
ARC consortium (NorduGrid, NDGF, KnowARC etal), together with gLite and Unicore, contribute tocreation of the Universal Middleware Distribution(UMD)for the European Grid Initiative (EGI) Sites and VOs that use ARC will get an access to the
European e-Science infrastructure, just like those thatuse gLite or Unicore
2009-03-02 14www.ndgf.org www.nordugrid.org
7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure
15/15
2006 Open Grid Forum
Summary: how to proceed
Get documentation at http://www.nordugrid.org Useful tips also at http://www.ndgf.org, and new ideas at http://www.knowarc.eu
Get Grid certificates for you, your hardware and your users
Get ARC: http://download.nordugrid.org official source and binary packages, some 3rd
party middleware
https://svn.nordugrid.org code, arc0(production) and arc1 (WS) directories ofthe nordugridrepository
Submit support requests: support@ndgf.org NDGF infrastructure and operations
nordugrid-support@nordugrid.org ARC middleware installation and usage
http://bugzilla.nordugrid.org feature requests, bug reports
Join the community:
Sign up for the nordugrid-discuss@nordugrid.org mailing list Attend workshops and conferences
Read ARC reference paper: "Advanced Resource Connector middleware for lightweight computational
Grids". M.Ellert et al., Future Generation Computer Systems 23 (2007) 219-240.
2009-03-02 15www.ndgf.org www.nordugrid.org
http://www.nordugrid.org/http://www.ndgf.org/http://www.knowarc.eu/http://download.nordugrid.org/https://svn.nordugrid.org/mailto:support@ndgf.orgmailto:nordugrid-support@nordugrid.orghttp://bugzilla.nordugrid.org/mailto:nordugrid-discuss@nordugrid.orgmailto:nordugrid-discuss@nordugrid.orgmailto:nordugrid-discuss@nordugrid.orgmailto:nordugrid-discuss@nordugrid.orghttp://bugzilla.nordugrid.org/mailto:nordugrid-support@nordugrid.orgmailto:nordugrid-support@nordugrid.orgmailto:nordugrid-support@nordugrid.orgmailto:support@ndgf.orghttps://svn.nordugrid.org/http://download.nordugrid.org/http://www.knowarc.eu/http://www.ndgf.org/http://www.nordugrid.org/
top related