www.eu-eela.eu JRA1 – Application and Infrastructure Grid Services Francisco Brasileiro Universidade Federal de Campina Grande – UFCG (Brazil) Diego Scardaci Istituto Nazionale di Fisica Nucleare – INFN (Italy) EELA-2 1 st Conference: All Members Meeting Bogotá, 24.02.2009
27
Embed
Www.eu-eela.eu JRA1 – Application and Infrastructure Grid Services Francisco Brasileiro Universidade Federal de Campina Grande – UFCG (Brazil) Diego Scardaci.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
www.eu-eela.eu
JRA1 – Application and Infrastructure Grid Services
Francisco Brasileiro
Universidade Federal de Campina Grande – UFCG (Brazil)
Diego Scardaci
Istituto Nazionale di Fisica Nucleare – INFN (Italy)
EELA-2 1st Conference: All Members Meeting
Bogotá, 24.02.2009
www.eu-eela.eu
• Agenda–Role of the JRA1–Services developed in the context of
the activity–How to get help
2Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
• JRA1 Goals–Help in fostering the sustainability of the
e-InfrastructureMaking the e-Infrastructure more interesting
and wide spread by increasing its reach and its usability
Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 3
www.eu-eela.eu
• Increase the reach of the e-infrastructure by:– Allowing the scavenging of idle resources
Create the necessary mechanisms to allow resource centres that run the OurGrid middleware to co-exist with resource centres running gLite within the EELA platform
Provide some level of interoperation between these different kinds of resource centres and their associated applications
– Allowing the execution of the grid middleware on top of platforms other than Scientific Linux Port the gLite middleware to the Windows platform Leveraging on the multi-platform
characteristics of OurGrid
Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 4
www.eu-eela.eu
• Increase the usability of the e-infrastructure by:– Developing new application-oriented grid
services
– Leveraging the grid services provided by the OurGrid middleware to execute bag-of-tasks jobs
– Facilitating the management of resource centres
Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 5
www.eu-eela.eu
• Services developed– Infrastructure-oriented
The main development here allows the co-existence of service grids (gLite-based) with opportunistic grids (OurGrid-based)
• only one super interface for Data Grid Application
• modular and extensible
13Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
1° Use Case: DMS Web Interface
14Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
Transactional GSAF
• Transaction Manager– Provides developers with a Transaction Pattern
autocommit(), execute(), commit(), rollback() performs several data manipulation in Atomic Mode
– Hold Runtime Exceptions to limit Data Inconsistence– System Crash Recovery to avoid Data Loss
• Challenge– SOA of Grid middleware makes transactions impossible at
server side– GSAF works at client side
Not full access to resources and services (locks) Not cache mechanism to save temporary data (memory) Not backup system to store previous data status (restore) Not full access control to distributed resources (centralization) …..
15Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
State of the Art
• GSAF – Framework released– Adopters
ADAT Project (Archdiocese of Catania): Cultural Heritage Digital Archive (antique manuscripts)
BM Portal project (Bio-Lab, DIST University of Genoa ) :Service platform to deploy and delivery bio medical and bio informatic applications
Aiuri project (COPPE/UFRJ - BRAZIL):aims to implement a Grid Oriented platform to support data and text mining applications uses GS.
• T-GSAF– Analysis and Design (Completed!)– Development, testing, production (Almost DONE!)
16Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
Links
• GSAF on EELA-2 forge:– https://forge.eu-eela.eu/projects/gsaf/
• GSAF on EELA-2 wiki:– https://grid.ct.infn.it/twiki/bin/view/EELA2/GSAF
17Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
A Secure Storage Service for the gLite Middleware
Diego Scardaci – INFN Catania
Giordano Scuderi – UNICO S.R.L.
18Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
Insider Abuse: the problem
• A grid user could store sensitive data in a Storage Elements managed by external organizations.
• Storage Elements Administrators could access data (but the data are sensitive!). For this reason data MUST be stored in an encrypted format.
• Data Encryption/Decryption MUST be performed inside user secure environment (for example inside the user’s organization).
19Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
A Secure Storage service for the gLite Middleware
• Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way.
The service is composed by the following components:• Command Line Applications: commands integrated in
the gLite User Interface to encrypt/upload and decrypt/ download files.
• Application Program Interface: allows the developer to write programs able to manage confidential data .
• Keystore: a new grid element used to store and retrieve the users’ keys.
20Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
Links
• Secure Storage on EELA-2 forge:– https://forge.eu-eela.eu/projects/securestorage/
• Secure Storage on EELA-2 wiki:– https://grid.ct.infn.it/twiki/bin/view/EELA2/SecureStorage
21Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
WatchDog Monitoring and controlling job execution
on the WorkerNode
Riccardo Bruno
INFN – Catania
22Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
Why do we need a WatchDog ?
• Why– Especially long term jobs require to be monitored and controlled during their
execution.• How
– Perform job control and monitoring using grid services in the less invasive way.• Observations
– Almost all jobs submitted on the grid are piloted by shell scripts Shell scripting allow to get precious info in case of faults Shell scripting can pilot more complex batch execution
– Both AMGA and SE+LFC can be used as the simplest IS on the grid. lfc-* and lcg-* tools already available for file creation and retrieve The latency of CLI tools for the storage is very low compared to long term jobs
• Requirements:– Monitor job execution watching snapshot of files produced by the job execution
File snapshot will be reported on LFC+SE or AMGA servers– It would be useful to configure the monitoring tool accordingly to the user needs
Few shell environment variables can be used to configure the watchdog tool– Control the job execution accessing directly on the WN
It is possible to send commands on the WN23Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
How does the WatchDog work ?
• The Watchdog is a shell script to be included in the JDL main script.
– Some watchdog features: It starts in background before to run the
long term job The watchdog runs as long as the main job The main script can control, stop and wait
until the watchdog has finished Easily and highly configurable and
customizable The watchdog does not compromise the
CPU power of the WN Amga LFC+SE
Commandsor Scripts
JDL
WN
File snapshots
WN
WN
UI
UI UI
24Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009
www.eu-eela.eu
Links
• Watchdog on EELA-2 forge:– https://forge.eu-eela.eu/projects/watchdog/
• WatchDog on EELA-2 wiki:– https://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog