The EU DataGrid
Post on 15-Jan-2016
47 Views
Preview:
DESCRIPTION
Transcript
The EU DataGrid
The European DataGrid Project Team
http://www.eu-datagrid.org/
Erwin.Laure@cern.ch
The EDG Intro– Tutorial - n° 2
Tutorial Roadmap
Project Introduction
Security Architecture
The EDG Testbed
Coffee Break
Specific Middleware Issues Job Management
Data Management
Monitoring & Fabric Management
Application Examples
The EU DataGrid Project Introduction
The European DataGrid Project Team
http://www.eu-datagrid.org/
The EDG Intro– Tutorial - n° 5
Contents
The EDG Project scope
Achievements
EDG structure
Middleware Workpackages: Goals, Achievements
DataGrid in Numbers
Relation to Sister Projects
The EDG Intro– Tutorial - n° 6
The Grid Vision
Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource
From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”
Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of…
central location,
central control,
omniscience,
existing trust relationships.
The EDG Intro– Tutorial - n° 7
Grids: Elements of the Problem
Resource sharing Computers, storage, sensors, networks, …
Sharing always conditional: issues of trust, policy, negotiation, payment, …
Coordinated problem solving Beyond client-server: distributed data analysis, computation,
collaboration, …
Dynamic, multi-institutional virtual orgs Community overlays on classic org structures
Large or small, static or dynamic
The EDG Intro– Tutorial - n° 8
Goals DataGrid is a project funded by European Union whose objective is to
exploit and build the next generation computing infrastructure providing intensive computation and analysis of shared large-scale databases.
Enable data intensive sciences by providing world wide Grid test beds to large distributed scientific organizations ( “Virtual Organizations, Vos”)
Start ( Kick off ) : Jan 1, 2001 End : Dec 31, 2003
Applications/End Users Communities : HEP, Earth Observation, Biology
Specific Project Objectives: Middleware for fabric & grid management Large scale testbed Production quality demonstrations Collaborate and coordinate with other projects (Globus, Condor, CrossGrid,
DataTAG, etc) Contribute to Open Standards and international bodies
( GGF, Industry&Research forum)
The EDG Intro– Tutorial - n° 9
DataGrid Main Partners
CERN – International (Switzerland/France)
CNRS - France
ESA/ESRIN – International (Italy)
INFN - Italy
NIKHEF – The Netherlands
PPARC - UK
The EDG Intro– Tutorial - n° 10
Research and Academic Institutes•CESNET (Czech Republic)•Commissariat à l'énergie atomique (CEA) – France•Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI)•Consiglio Nazionale delle Ricerche (Italy)•Helsinki Institute of Physics – Finland•Institut de Fisica d'Altes Energies (IFAE) - Spain•Istituto Trentino di Cultura (IRST) – Italy•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany•Royal Netherlands Meteorological Institute (KNMI)•Ruprecht-Karls-Universität Heidelberg - Germany•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands•Swedish Research Council - Sweden
Assistant Partners
Industrial Partners•Datamat (Italy)•IBM-UK (UK)•CS-SI (France)
The EDG Intro– Tutorial - n° 11
Project Schedule Project started on 1/Jan/2001
Testbed 0 (early 2001) International test bed 0 infrastructure deployed
Globus 1 only - no EDG middleware
Testbed 1 ( 2002 ) First release of EU DataGrid software to defined users within the project:
HEP experiments (WP 8), Earth Observation (WP 9), Biomedical applications (WP 10)
Testbed 2 (End 2002) Builds on Testbed 1 to extend facilities of DataGrid
Focus on production quality
Testbed 3 (2003) Advanced functionality; currently being deployed.
Project stops on 31/Dec/2003
The EDG Intro– Tutorial - n° 12
DataGrid Work Packages The EDG collaboration is structured in 12 Work Packages
WP1: Work Load Management System WP2: Data Management WP3: Grid Monitoring / Grid Information Systems WP4: Fabric Management WP5: Storage Element WP6: Testbed and demonstrators – Production quality
International Infrastructure WP7: Network Monitoring WP8: High Energy Physics Applications WP9: Earth Observation WP10: Biology WP11: Dissemination WP12: Management
The EDG Intro– Tutorial - n° 13
DataGrid Architecture
Collective ServicesCollective Services
Information &
Monitoring
Information &
Monitoring
Replica ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault Tolerance
Resource Management
Resource Management
Fabric StorageManagement
Fabric StorageManagement
Grid
Fabric
Local Computing
Grid Grid Application LayerGrid Application Layer
Data Management
Data Management
Job Management
Job Management
Metadata Management
Metadata Management
Object to File
Mapping
Object to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
The EDG Intro– Tutorial - n° 14
EDG Interfaces
Collective ServicesCollective Services
Information & MonitoringInformation
& MonitoringReplica ManagerReplica Manager
Grid Scheduler
Grid Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
Computing Computing ElementsElements
SystemSystemManagersManagers
ScientiScientistssts
OperatingOperatingSystemsSystems
FileFile SystemsSystems
StorageStorageElementsElementsMassMass Storage Storage
SystemsSystemsHPSS, CastorHPSS, Castor
UserUser AccountsAccounts
CertificateCertificate AuthoritiesAuthorities
ApplicatiApplicationonDevelopeDevelopersrs
BatchBatch SystemsSystemsPBS, LSFPBS, LSF
The EDG Intro– Tutorial - n° 15
WP1: Work Load Management
Goals Maximize use of resources by
efficient scheduling of user jobs
Achievements Definition of architecture for
scheduling & res. mgmt. and accounting & reservation
Development of "super scheduling" component using application data and computing elements requirements
Support for MPI jobs Logical job check pointing Interactive jobs
Collective ServicesCollective Services
Information & Monitoring
Information & Monitoring
Replica ManagerReplica Manager
Grid Scheduler
Grid Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & BookkeepingLogging &
Bookkeeping
The EDG Intro– Tutorial - n° 16
EDG middleware architecture: The Workload Management System
(WP1)
WP1 is responsible for the Workload Management System (WMS).
The WMS is currently composed by the following parts:
User Interface (UI) : access point for the user to the GRID ( using JDL)
Resource Broker (RB) : the broker of GRID resources, matchmaking
Job Submission System (JSS) : Condor-G; interfacing batch systems
Information Index (II) : an LDAP server used as a filter to select resources
Logging and Bookkeeping services (LB) : MySQL databases to store Job Info
The EDG Intro– Tutorial - n° 17
WP1: Work Load Management
ComponentsJob Description Language
Resource Broker
Job Submission Service
Information Index
User Interface
Logging & Bookkeeping Service
Collective ServicesCollective Services
Info & MonitorInfo &
MonitorReplica
ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication
Accounting
Authorization Authentication
Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
Fabric servicesFabric services
ConfigManagement
ConfigManagement
Node Installation Management
Node Installation Management
MonitoringFault
Tolerance
MonitoringFault
Tolerance
Resource Managem.Resource
Managem.Fabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Managem.
Data Managem.
Metadata Managem.Metadata
Managem.Object to
File Mapping
Object to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
Implementation: UI : python (LB client : C++)
RB : C++
JSS : C++, python
II : LDAP server
LB: MySQL, C++
Input/Output Sandboxes: GridFTP
Job Managem.
Job Managem.
SQL Database Services
SQL Database Services
WMS main interfaces: Globus Gatekeeper
WP2 Replica Catalog APIs
WP3 Information Systems
WP7 network monitoring info providers
End User (using JDL files, on the UI)
The EDG Intro– Tutorial - n° 18
WP2: Data Management Goals
Coherently manage and share petabyte-scale information volumes in high-throughput production-quality grid environments
Achievements Survey of existing tools and technologies
for data access and mass storage systems
Definition of architecture for data management
Deployment of Grid Data Mirroring Package (GDMP) in Testbed 1
Deployment of EDG Replica Manager in Testbed 2
Close collaboration with Globus, PPDG/GriPhyN & Condor
Common design of RLS Working with GGF on standards
Collective ServicesCollective Services
Information & Monitoring
Information & Monitoring
Replica ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & BookkeepingLogging &
Bookkeeping
The EDG Intro– Tutorial - n° 19
EDG middleware architecture: WP2 (Data Management )
WP2 is responsible for Data Management, which includes file and replica management, metadata access and data security.
WP2 components:
Replica Manager: the main manager for triggering replica execution all over the GRID, including replica optimization and interfacing the replica catalog service
Replica Catalog: a GRID service used to resolve Logical File Names into a set of corresponding Physical File Names – Globus Replica Catalog and Replica Location Service (RLS)
GDMP: the GRID Data Mirroring Package, used to create replicas of any filetype all over the GRID Storage Elements in a synchronized way, by automatic updating the replica catalog
Spitfire: provides a Grid enabled middleware service for access to relational databases : it consists of the Spitfire Server module and the Spitfire Client libraries and command line executables.
The EDG Intro– Tutorial - n° 20
WP2: Data Management
Deployed ComponentsGridFTP
Replica Manager - edg-replica-manager and Reptor
Replica Catalog - globus-replica-catalog
GDMP
Spitfire
Collective ServicesCollective Services
Info & MonitorInfo &
MonitorGrid
SchedulerGrid
SchedulerReplica
ManagerReplica
Manager
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication
Accounting
Authorization Authentication
Accounting
Replica CatalogReplica Catalog
Fabric servicesFabric services
ConfigManagement
ConfigManagement
Node Installation Management
Node Installation Management
MonitoringFault
Tolerance
MonitoringFault
Tolerance
Resource Managem.Resource
Managem.Fabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Job Managem.
Job Managem.
Metadata Managem.Metadata
Managem.Object to
File Mapping
Object to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
Implementation: RM: C++
Reptor: Java based Web Services
RC : Globus Replica Catalog wrapper
GDMP : C++
Spitfire : Java, Web Services
Data Managem.
Data Managem.
SQL Database Services
SQL Database Services
WP2 main interfaces: The GRID Storage Element
WP1 Resource Broker APIs
WP3 GRID Info services
WP7 network monitoring info providers
End User (using GDMP)
Storage Element Services
Storage Element Services
The EDG Intro– Tutorial - n° 21
WP3: Grid Monitoring Services
Goals Provide information system for
discovering resources and monitoring status
Achievements Survey of current technologies Coordination of schemas in testbed 1 Development of Ftree caching backend
based on OpenLDAP (Light Weight Directory Access Protocol) to address shortcoming in MDS v1
Relational Grid Monitoring Architecture (R-GMA)
GRM and PROVE adapted to grid environments to support end-user application monitoring
Collective ServicesCollective Services
Information & Monitoring
Information & Monitoring
Replica ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorizat ion Authentication and Accounting
Authorizat ion Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & Book-keepingLogging &
Book-keeping
The EDG Intro– Tutorial - n° 22
WP3’s task is to provide information about
The Grid itself This includes information about resources (ComputingElements, StorageElements and the Network), for which the Globus MDS is a common solution; and job status information(as implemented by WP1's Logging and Bookkeeping).
Grid applications This is information published by user jobs. This is used for performance monitoring.
R-GMA relational implementation of the GGF GMA
interoperable with MDS
WP3 : GRID Monitoring and Info Providers
The EDG Intro– Tutorial - n° 23
WP3: GRID Monitoring
ComponentsMDS / FTree
R-GMA
GRM/Prove
Collective ServicesCollective Services
Info & MonitorInfo &
MonitorReplica
ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication
Accounting
Authorization Authentication
Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
Fabric servicesFabric services
ConfigManagement
ConfigManagement
Node Installation Management
Node Installation Management
MonitoringFault
Tolerance
MonitoringFault
Tolerance
Resource Managem.Resource
Managem.Fabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Managem.
Data Managem.
Metadata Managem.Metadata
Managem.Object to
File Mapping
Object to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
Implementation: MDS : LDAP, Globus GRIS, GIIS
FTree : OpenLDAP, caching
R-GMA : Java , C++, MySQL, TomCat
GRM / PROVE : P-GRADE
Job Managem.
Job Managem.
SQL Database Services
SQL Database Services
WP3 main interfaces: WP1 Resource Broker ( InfoIndex)
WP2 RM optimizer
all GRID services producing info (SE,CE..)
WP7 network monitoring
The EDG Intro– Tutorial - n° 24
WP4: Fabric Management Goals
manage clusters (~thousands) of nodes
Achievements Survey of existing tools,
techniques and protocols Defined an agreed architecture
for fabric management Initial implementations deployed
at several sites in testbed 1 & 2
Collective ServicesCollective Services
Information & Monitoring
Information & Monitoring
Replica ManagerReplica Manager
Grid Scheduler
Grid Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & Book-keepingLogging &
Book-keeping
The EDG Intro– Tutorial - n° 25
WP4 is responsible to deliver a computing fabric comprised of all the necessary tools to manage a center providing grid services on clusters of thousands of nodes. The computing fabric is called the Computing Element in EDG.
User Job Control and Management (Grid and local jobs) on fabric batch and/or interactive CPU services
Gridification – Grid interface to fabric resources
Resource Management – manage underlying batch services
Automated System Administration for Computing Fabric Elements. These subsystems are reserved for system administrators and operators for performing system maintenance
Configuration Management
Installation Management
Fabric Monitoring
EDG middleware architecture: WP4 : Fabric Management
Components
The EDG Intro– Tutorial - n° 26
WP4: Fabric Management
ComponentsLCFG
Fabric Monitoring
PBS & LSF info providers
Image installation
Config. Cache Mgr
Collective ServicesCollective Services
Info & MonitorInfo &
MonitorReplica
ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication
Accounting
Authorization Authentication
Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
Fabric servicesFabric services
ConfigManagement
ConfigManagement
Node Installation Management
Node Installation Management
MonitoringFault
Tolerance
MonitoringFault
Tolerance
Resource Managem.Resource
Managem.Fabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Managem.
Data Managem.
Metadata Managem.Metadata
Managem.Object to
File Mapping
Object to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
Implementation: LCFG : C++, XML, HTTP
Job Managem.
Job Managem.
SQL Database Services
SQL Database Services
WP4 main interfaces: WP1 Resource Broker ( InfoIndex)
WP2 Data management
WP5 Storage Element
WP3 GRID Info Services
The EDG Intro– Tutorial - n° 27
WP5: Mass Storage Management Goals
Provide common user and data export/import interfaces to existing local mass storage systems
Achievements Review of Grid data systems, tape
and disk storage systems and local file systems
Definition of Architecture and Design for DataGrid Storage Element
Collaboration with Globus on GridFTP/RFIO
Collaboration with PPDG on control API
First attempt at exchanging Hierarchical Storage Manager (HSM) tapes
SRM compliant interface to MSS
Collective ServicesCollective Services
Information & Monitoring
Information & Monitoring
Replica ManagerReplica Manager
Grid Scheduler
Grid Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & BookkeepingLogging &
Bookkeeping
The EDG Intro– Tutorial - n° 28
WP5 delivers the Grid interface to Storage.
Its service, the Storage Element (SE) is interfacing to underlying Mass Storage Systems or simple storage services.
Main interfaces: Data, gridftp will be used to transfer files over the WAN and
the files will optionally be available to local nodes by NFS.
Information, Existing MDS information providers will be extended to provide the extra information in the GLUE storage schema.
Control, functions such as reservation, pinning, deletion, and transfer time estimation. Will provide an SRM 2 interface.
WP5 : Mass Storage Management
The EDG Intro– Tutorial - n° 29
WP5: Mass Storage Management Achievements
Definition of Architecture and Design for DataGrid storage Element
Collaboration with Globus on GridFTP/RFIO
Collaboration with PPDG on control API Staging from/to CASTOR at CERN
succesfully implemented and tested Succesfully Interfaced to GDMP
Supported Storage Systems: UNIX disk systems HPSS (High Performance Storage
System) CASTOR (through RFIO) GridFTP servers DMF Enstore
Collective ServicesCollective Services
Info & MonitorInfo &
MonitorReplica
ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication
Accounting
Authorization Authentication
Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
Fabric servicesFabric services
ConfigManagement
ConfigManagement
Node Installation Management
Node Installation Management
MonitoringFault
Tolerance
MonitoringFault
Tolerance
Resource Managem.Resource
Managem.Fabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Managem.
Data Managem.
Metadata Managem.Metadata
Managem.Object to
File Mapping
Object to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
Job Managem.
Job Managem.
SQL Database Services
SQL Database Services
WP5 (SE) main interfaces: WP1 Resource Broker & JSS
WP2 RM, RC
WP7 for GRIDftp monitoring
WP3 GRID Info Services
The EDG Intro– Tutorial - n° 30
WP6: TestBed Integration Goals
Deploy testbeds for the end-to-end application experiments & demos
Integrate successive releases of the software components
Achievements Integration of EDG sw and deployment
Working implementation of multiple Virtual Organizations (VOs) s & basic security infrastructure
Definition of acceptable usage contracts and creation of Certification Authorities group
Definition of test plan
User’s, administrator’s, and developer’s guides
Components
Globus packaging & EDG config
Build tools
End-user documents
Collective ServicesCollective Services
Information & Monitoring
Information & Monitoring
Replica ManagerReplica Manager
Grid Scheduler
Grid Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & BookkeepingLogging &
Bookkeeping
WP6 additionsto Globus
GlobusEDG release
The EDG Intro– Tutorial - n° 31
Tasks for the WP6 integration team Testing and integration of the Globus package
Exact definition of RPM lists (components) for the various testbed machine profiles (CE service , RB, UI, SE service , NE, WN, ) – check dependencies
Perform preliminary centrally (CERN) managed tests on EDG m/w before green light for spread EDG testbed sites deployment
Provide, update end user documentation for installers/site managers, developers and end users
Define EDG release policies, coordinate the integration team staff with the various WorkPackage managers – keep high inter-coordination.
Assign the reported bugs to the corresponding developers/site managers (BugZilla)
Complete support for the iTeam testing VO
The EDG Intro– Tutorial - n° 32
WP6: TestBed Integration and demonstrators
WP6 goals: the EDG testbed Integration of EDG sw releases and
deployment all over the EDG testbed : the integration team
Working implementation of multiple VOs & basic security infrastructure
Definition of acceptable usage contracts and creation of Certification Authorities group
Set up of the Authorization Working Group to manage authorization policies on the testbed
2 Testbeds: Dev. TB for integration Application TB for application usage Certification TB planned
Components
Support for test-VO, mkgridmap tools
Globus packaging & EDG config
Build tools, CVS central s/w repository
End-user documents
Collective ServicesCollective Services
Info & MonitorInfo &
MonitorReplica
ManagerReplica
ManagerGrid
SchedulerGrid
Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication Accounting
Authorization Authentication Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
Fabric servicesFabric services
ConfigManagement
ConfigManagement
Node Installation Management
Node Installation Management
MonitoringFault
Tolerance
MonitoringFault
Tolerance
Resource Managem.Resource
Managem.Fabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Managem.
Data Managem.
Metadata Managem.Metadata
Managem.Object to
File Mapping
Object to File
Mapping
Logging & Book-
keeping
Logging & Book-
keeping
Job Managem.
Job Managem.
SQL Database Services
SQL Database Services
The EDG Intro– Tutorial - n° 33
WP7: Network Services Goals
Review the network service requirements for DataGrid
Establish and manage the DataGrid network facilities
Monitor the traffic and performance of the network
Deal with the distributed security aspects
Achievements Analysis of network requirements for testbed
1 & study of available network physical infrastructure
Use of European backbone GEANT since Dec. 2001
Initial network monitoring architecture defined and first tools deployed
Collaboration with Dante & DataTAG Working with GGF (Grid High Performance
Networks) & Globus (monitoring/MDS) Network cost estimation for workload and
data management
Components
network monitoring tools:
PingER
Udpmon
Iperf
Collective ServicesCollective Services
Information & Monitoring
Information & Monitoring
Replica ManagerReplica Manager
Grid Scheduler
Grid Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication and Accounting
Authorization Authentication and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault ToleranceResource
ManagementResource
ManagementFabric StorageManagement
Fabric StorageManagement
Grid Application LayerGrid Application Layer
Data Management
Data ManagementJob
ManagementJob
ManagementMetadata
ManagementMetadata
ManagementObject to File
MappingObject to File
Mapping
Logging & BookkeepginLogging &
Bookkeepgin
The EDG Intro– Tutorial - n° 34
Applications (WP8-10)
High Energy Physics
Biomedical Applications
Earth Observation Science Applications
The EDG Intro– Tutorial - n° 35
Grid aspects covered by EDG
VO servers LDAP directory for mapping users (with certificates) to correct VO
Storage Element Grid-aware storage area, situated close to a CE
User Interface Submit & monitor jobs, retrieve output
Replica Manager Replicates data to one or more CEs
Job Submission Service
Manages submission of jobs to Res. Broker
Replica Catalog Keeps track of multiple data files “replicated” on different CEs
Information index Provides info about grid resources via GIIS/GRIS hierarchy
Information & Monitoring
Provides info on resource utilization & performance
Resource Broker Uses Info Index to discover & select resources based on job requirements
Grid Fabric Mgmt Configure, installs & maintains grid sw packages and environ.
Logging and Bookkeeping
Collects resource usage & job status
Network performance, security and monitoring
Provides efficient network transport, security & bandwidth monitoring
Computing Element Gatekeeper to a grid computing resource
Testbed admin. Certificate auth.,user reg., usage policy etc.
The EDG Intro– Tutorial - n° 36
Software
50 use cases
18 software releases
>300K lines of code
People
>350 registered users
12 Virtual Organisations
16 Certificate Authorities
>200 people trained
278 man-years of effort
100 years funded
DataGrid in Numbers
Testbeds
>15 regular sites
>10’000s jobs submitted
>1000 CPUs
>5 TeraBytes disk
3 Mass Storage Systems
Scientific applications5 Earth Obs institutes9 bio-informatics apps6 HEP experiments
The EDG Intro– Tutorial - n° 37
Through links with sister projects, there is thepotential for a truely global scientific applications grid
Demonstrated at IST2002 and SC2002 in November
Related Grid Projects
top related