Parallel and Distributed Scilab Simulations with ProActive ... · Parallel and Distributed Scilab Simulations with ProActive Parallel Suite Parallelism is Complex: ... Video 1: IC2D

Agenda

1. Background: INRIA, ActiveEon

2. ProActive Open Source: Programming, Scheduling, Resourcing

3. Parallel Scilab Simulations

4. ANR OMD2: Distributed Multi-Disciplinary Optimizations

5. Conclusion: Cloud Portal ProActive PACA Cloud

D. Caromel, et al.

Parallel and Distributed

Scilab Simulations with

ProActive Parallel Suite

Parallelism is Complex:

Hiding Details and Seamless Integration with Scilab

2 2

1. Background

1. Background

3 3

OASIS Team & INRIA

A joint team, Now about 35 persons

2004: First ProActive User Group

2009, April: ProActive 4.1, Distributed & Parallel:

From Multi-cores to Enterprise GRIDs

4 4

OASIS Team Composition (35)

Researchers (5): D. Caromel (UNSA, Det. INRIA)

E. Madelaine (INRIA)

F. Baude (UNSA)

F. Huet (UNSA)

L. Henrio (CNRS)

PhDs (11): Antonio Cansado (INRIA, Conicyt)

Brian Amedro (SCS-Agos)

Cristian Ruz (INRIA, Conicyt)

Elton Mathias (INRIA-Cordi)

Imen Filali (SCS-Agos / FP7 SOA4All)

Marcela Rivera (INRIA, Conicyt)

Muhammad Khan (STIC-Asia)

Paul Naoumenko (INRIA/Région PACA)

Viet Dung Doan (FP6 Bionets)

Virginie Contes (SOA4ALL)

Guilherme Pezzi (AGOS, CIFRE SCP)

+ Visitors + Interns

PostDoc (1): Regis Gascon (INRIA)

Engineers (10):

Elaine Isnard (AGOS)

Fabien Viale (ANR OMD2, Renault )

Franca Perrina (AGOS)

Germain Sigety (INRIA)

Yu Feng (ETSI, FP6 EchoGrid)

Bastien Sauvan (ADT Galaxy)

Florin-Alexandru.Bratu (INRIA CPER)

Igor Smirnov (Microsoft)

Fabrice Fontenoy (AGOS)

Open position (Thales)

Trainee (2): Etienne Vallette d’Osia (Master 2 ISI)

Laurent Vanni (Master 2 ISI)

Assistants (2): Patricia Maleyran (INRIA)

Sandra Devauchelle (I3S) Located in Sophia Antipolis, between

Nice and Cannes, Visitors Welcome!

5 5

Co-developing, Support for ProActive Parallel Suite

Worldwide Customers: Fr, UK, Boston USA

Startup Company Born of INRIA

Some Partners:

Some Customers:

6

2. ProActive Parallel Suite

6

7

Product: ProActive Parallel Suite

Java Parallel

Toolkit

Multi-Platform

Job Scheduler

Resource

Manager

Strong Differentiation:

Java Parallel Programming + Integration + Portability: Linux, Windows, Mac +

Versatility: Desktops, Cluster, Grid, Clouds = Perfect Flexibility

Used in Production Today:

50 Cores 300 Cores early 2010

8 8

ProActive Programming:

Active Objects

9 9

ProActive Programming View

GPU nodes

10 10

ProActive Programming View

10

GPU nodes

11 11

TYPED

ASYNCHRONOUS GROUPS

12 12 12

Broadcast and Scatter

JVM

JVM

JVM

JVM

ag cg

ag.bar(cg); // broadcast cg

ProActive.setScatterGroup(cg);

ag.bar(cg); // scatter cg

c1 c2 c3 c1 c2 c3

c1 c2 c3 c1 c2 c3

c1 c2 c3

c1 c2 c3

s

c1 c2 c3

s

Broadcast is the default behavior

Use a group as parameter, Scattered depends on rankings

13 13 13

Dynamic Dispatch Group

JVM

JVM

JVM

JVM

ag cg

c1

c2

c3

c4

c5

c6

c7

c8 c0

c9 c1

c2

c3

c4

c5

c6

c7

c8 c0

c9

c1

c2

c3

c4

c5

c6

c7

c8 c0

c9

Slowest

Fastest

ag.bar(cg);

Abstractions

for Parallelism

The right Tool to do the Task right

15 15

ProActive Parallel Suite

Workflows in Java

Master/Workers

SPMD

Components …

16 16

Object-Oriented

SPMD

Key Point

“MPI and programming languages from

the 60’s will not make it”

Jack Dongarra, 2/13/2009,

Wake Forest University talk

“It is time to get ride of MPI”

Alan Edelman, MIT, 06/16/2010,

ScilabTec’10 Users’ Day

18 18 18

OO SPMD: Object-Oriented SPMD A ag = newSPMDGroup (“A”, […], VirtualNode)

// In each member

myGroup.barrier (“2D”); // Global Barrier

myGroup.barrier (“vertical”); // Any Barrier

myGroup.barrier (“north”,”south”,“east”,“west”);

A

19 19

IC2D: Optimizing

20 20

IC2D

21 21

IC2D

22 22

ChartIt

23 23

Pies for Analysis and Optimization

24

Video 1:

IC2D Optimizing

Monitoring, Debugging, Optimizing

25 25

Scheduling & Resourcing

26 26

ProActive Scheduling

26

27

ProActive Scheduling Big Picture

RESOURCES

Multi-platform Graphical Client (RCP)

File-based or LDAP authentication

Static Workflow Job Scheduling, Native and

Java tasks, Retry on Error, Priority Policy, Configuration Scripts,…

Dynamic and Static node sources, Resource

Selection by script, Monitoring and Control GUI,…

ProActive Deployment capabilities:

Desktops, Clusters, Clouds,…

ProActive

Scheduler

ProActive

Resource Manager

28

Job

Workflow Example : Picture Denoising

Split

Denoise Denoise Denoise Denoise

Merge

•with selection on native executable availability (ImageMagik, GREYstoration)

• Multi-platform selection and command generation •with file transfer in pre/post scripts

29 29

ProActive Resourcing

29

30

RESOURCING User Interface

30

31 31

Clusters to Grids to Clouds:

e.g. on Amazon EC2

32

Node source Usecase :

Configuration for external cloud with EC2

ProActive

Scheduler

ProActive

Resource Manager

Dedicated resources

LSF

Static Policy

Amazon EC2

EC2

Dynamic

Workload Policy

Desktops

Desktops

Timing Policy

12/24

33

Video 2:

Scheduler, Resource Manager

34

3.

Parallel Scilab Simulations

34

35

Seamless Parallel & Distributed Scilab

Dedicated resources

LSF

Static Policy

Amazon EC2

EC2

Dynamic

Workload Policy

Desktops

Desktops

Timing Policy

12/24

36

Interface ProActive Scilab

37

Interface ProActive Matlab

38

Interface ProActive Scilab / Matlab

Primitives de haut niveau Scilab/Matlab pour distribuer les calculs :

f = PAeval(@foo,p1) // evalue f(p1) en asynchrone // f est un futur

L = PAsolve(@foo, {p1,p2, … pk}) // evalue foo(p1), foo(p2) .. foo(pk) en asynchrone // L = f1,…fk est une liste de futurs

39

Synchronization des calculs

Primitives d’attente :

r = PAwaitForAny(L)

// bloque et attend jusqu’a l’obtention du

// premier des fk et retourne le resultat r

lr = PAgetAny(L)

// non-blocant, retourne une liste des

resultats // deja obtenus

40

4. Use Case:

OMD2

40

41

OMD2

Interfaces Open Sources

pour faciliter les

Optimisations Multi-Disciplinaires Distribuées

42

OMD2 :

Interfaces Open Sources

pour faciliter les

Optimisations Multi-Disciplinaires

Distribuées

43

Plateforme OMD2, vue générale :

Optimizer

(surrogate based, CMA ES ...)

Cluster 1 Cluster 2 Cluster N

Middleware Proactive

f(X1) Output

(Analysis)

adapco UTC INRIA

...

Sirehna +

CD-adapco

f(XN)

f(X2)

ENSM-SE, ECP TAO, UTC, ENS , INRIA

DIG

ITE

O /

SC

ILA

B

User /

Renault

test case 1

local

test cases 2,3,4

webservice

Input

(Xi,obj fct, constr)

end

44

Distribution des calculs

Scheduler

WS

BD

PO

Acquisition des maillages, etc

Stratégie de résolution

Boucle optimisation

Calculs f(x)

Mailleur Solveur

Soumission du PO

Catia

Visualisation stats, etc

N1

N2

Ni1

Nik

Ressources

45

Les cas tests

Conduit de climatisation 3D Conduit de climatisation 2D

Culasse multi-disciplinaire Aéro externe

<1min

CPU

10min

CPU

100h

CPU 1000h

CPU

46

Interface ProActive Scilab

47

Exemple d’utilisation : Boucle d’Optimisation

48

5.

ProActive PACA GRID

50

The ProActive PACA Grid Platform (4)

Total:

816 Cores

480 CUDA Cores

14.8TB Storage

Publically Available Today

51

ProActive in Cloud Stack

ProActive

Cloud Portal

IaaS

52

Conclusion

52

53

Conclusion

Java Parallel

Toolkit

Multi-Platform

Job Scheduler

Resource

Manager

Portability:

Windows, Linux, Mac

Versatility:

Desktops, Grids, Clouds

Free Professional

Open Source Software

Seamless Scilab Interface (& Matlab)

Multi-Core: Ready for a strong evolution

Cloud: Smooth transition available (Desktop , Server, Cluster)