Journée Présentation de lANR In conjunction with Perpi2006 RenPar'17 / SympA'2006 / CFSE'5 / JC'2006 3 octobre 2006.

Journée Journée Présentation de l’ANRPrésentation de l’ANR

In conjunction with Perpi’2006RenPar'17 / SympA'2006 / CFSE'5 / JC'20063 octobre 2006

2

Teams LIP/INRIA: Projet GRAAL

Anne Benoît Raphaël Bolze Yves Caniou Eddy Caron Pushpinder Kaur Chouhan Frédéric Desprez Jean-Sébastien Gay Cédric Tedeschi

IRISA/INRIA: Projet PARIS Gabriel Antoniu Luc Bougé Hinde Bouziane Loïc Cudennec Mathieu Jan Sébastien Monnet Christian Perez Thierry Priol

LaBRI/INRIA: Projet RUNTIME Olivier Aumage Alexandre Denis

ENSEEIHT: IRIT Michel Daydé Marc Pantel Daniel Hagimont

CERFACS Eric Maisonnave

ENS-Lyon: CRAL Hélène Courtois Julien Devriendt Romain Teyssier

3

Middleware

Components

Deployment Communications

Data management

Scheduling

The Concept

LEGO: League for Efficient Grid Operation.

Workflow

4

Components

Advanced Component Model

Components and data-sharing service Composition based on data-access

Data port Use of JuxMem

Components and master-worker paradigm Collection + request scheduling Use of Diet

Components and workflow What’s mean dependency for the component model?

Components and legacy code No code re-writing Mechanism to deal between application and scheduler

BA

Data

data_ref

workermaster

5

DIET Architecture

LA

MA

LA

LALA

Server front end

Master Agent

Local Agent

Client

MA

MA

MA

MA

CORBA or JXTA

QuickTime™ et undécompresseur TIFF (non compressé)

sont requis pour visionner cette image.



QuickTime™ et undécompresseur TIFF (non compressé)sont requis pour visionner cette image.


Middleware

6

Peer

Peer

Peer Peer

Peer

PeerPeer

Peer

PeerPeer

Peer

Peer

FirewallPeer

PeerTCP/IP

HTTP

Peer ID

Peer ID

Peer ID

Peer ID

Peer ID

Peer ID

Peer ID

Peer ID

Firewall

Toolbox for the development of P2P applications

Set of protocols

One peer Unique ID Several communication

protocols (TCP, HTTP, …)

JuxMem: a Grid Data-Sharing Service

A peer-to-peer architecture for a data-sharing service in memory

Persistence and data coherency mechanism Transparent data localization

Datamanagement

7

Communication Brick

Communication for multi-paradigm programming model Message passing Remote procedure calls Distributed/Shared memory

Cluster view: High-speed network Hardware heterogeneity

Myrinet, Quadrics, Infiniband, SCI Gigabit Ethernet

Software heterogeneity GM, MX Elan, Elan4 Sisci Sockets

Contribution Madeleine library

Communications

NetworkNetwork

ProgrammingenvironmentsProgrammingenvironments

Genericcommunication support

Genericcommunication support

Messagepassing

Serviceinvocation(RPC, RMI)

Madeleine

ApplicationprocessesApplicationprocesses

Ethernet Myrinet SCI Quadrics …

Distributedshared memory



8

Communications

Grid Communication: PadicoTM Grid communications between site

Wide communications Specific communications Connectivity: firewalls, none-routed network, etc. Performance: High latency, low bandwidth Security: protection,

accounting Middleware and applications

integration Middleware upgrading for

Madeleine? Existing code?

Contribution A high-performance

communication framework for Grids: PadicoTM

PARIS project (2000-2004) andRUNTIME (since 2004)







9

Scheduling Brick: into DIET

Plug-in Scheduler Existing plug-in scheduling facilities Application-specific definition of appropriate performance metrics An extensible measurement system Tuneable comparison/aggregation routines for scheduling Composite requirements enables various selection methods

basic resource availability processor speed, memory database contention future requests

CORI Collector: an easy interface to gathering performance and load

information for a specific SeD Two modules (currently): CoRI-Easy and FAST Possible to extend (new modules): Ganglia, Nagios, R-GMA,

Hawkeye, INCA, MDS, …

Scheduling

10

Scheduling Brick: Workflow Workflow management using

component model Workflow and DIET

Simple and high level API for the client Workflow description based on XML

Use of different scheduling algorithms (RR, HEFT, etc.)

Ability for the client to use its own workflow scheduler

Automatic rescheduling mechanisms Support multi-workflows scheduling DIET hierarchy extended with a special

agent: MADAG

Two execution modes of the MADAG Complete scheduling provided : task

priorities and resources mapping Partial scheduling provided : only task

priorities

Workflow

Exemple from Cosmological Application

11

Deployment Brick: ADAGE

Automatic deployment tool for grid environment

Only one command to deploy 3 kinds of input information Resource description

application description control parameter

Planning model (random, round-robin), …

Plug-in for each application Description convector Configuration of application

CCM, MPICH-P4, MPICH-G2, JXTA Plug-in: from 400 to 1200 C++ lines

Deployment

12

Ocean-atmosphere Numerical Simulations

Energy transport: Equator Pole World climate behavior Platform

Supercomputer approach large simulation (1000 years)

Grid approach parameterization design independent and simultaneous simulations

Code coupling ARPEGE v4.5 (atmospheric modelisation) OPA v9 +LIM (ocean modelisation) OASIS v3 (code coupling)

Application


13

Cosmological Simulation RAMSES

Computes the evolution of dark matter particles starting from the early universe's structure

GALICS Performs structure detection (halos of dark matter) Builds the evolution tree of the particles Generates galaxies

Application

Simulation 1st part, 1 submission from the client:

generating low resolution IC RAMSES post-processing with GALICS, results are sent

back to the user 2nd part, n submissions from the client:

generating high resolution IC centered on the wanted part of the universe

RAMSES post-processing with GALICS, results are sent

back to the user


QuickTime™ et undécompresseur Vidéo


14

Application

Sparse Direct Solvers Sparse direct solvers in a client-server

environment (DIET) Provide remote access to the algorithms we develop (e.g. MUMPS) Easy to use from a light client Data persistency on the servers is crucial

Application: an expertise site for sparse linear algebra: ACI GRID TLSE (coordinated by ENSEEIHT-IRIT, Toulouse)

On a user’s specific problem, compare execution time / accuracy / memory usage / … of various solvers:

public domain … as well as commercial, sequential … as well as parallel

Find best parameter values / reordering heuristics on a given problem Also bibliography, matrix collections, …

All elementary requests executed on the/a GRID through DIET Must be highly evolving (new solvers with new parameters, new

scenarii)



15

Conclusion

Programming model brick Component model

Grid middleware brick GridRPC environment: DIET.

Data management brick Data-sharing system: JuxMeM

Communication bricks Intra-cluster: Madeleine Grid communication: PadicoTM

Scheduling brick DIET’s Plug-in scheduler

Workflow bricks DIET’s DAG management Component management

Deployment brick ADAGE

Applications brick Ocean-atmosphere Numerical

Simulations Cosmological Simulation Sparse Direct Solvers

Questions?Questions?

http://graal.ens-lyon.fr/LEGO

QuickTime™ et undécompresseur TIFF (non compressé)sont requis pour visionner cette image.QuickTime™ et undécompresseur TIFF (non compressé)sont requis pour visionner cette image.





QuickTime™ et undécompresseur TIFF (non compressé)sont requis pour visionner cette image.QuickTime™ et undécompresseur TIFF (non compressé)sont requis pour visionner cette image.





Journée Présentation de lANR In conjunction with Perpi2006 RenPar'17 / SympA'2006 / CFSE'5 / JC'2006 3 octobre 2006.

Documents

scheduling slide

data data

cosmological application

grid data

scheduling brick

concept workflow slide

workflow workflow management

jxta middleware slide