A Chemistry-Inspired Workflow Management System for Scientific Applications on Clouds

A Chemistry-Inspired Workflow ManagementSystem for Scientific Applications in Clouds

Hector Fernandez, Cedric Tedeschi and Thierry Priol 00 MOIS 2011

7th IEEE International Conference on e–ScienceStockholm 2011

Context

• Scientific applications developed as workflows demanding more computational power. Demand for deployment on Grids or Clouds.

• Scientific workflow management systems (WMS): Implicit parallelism. Data-driven coordination. Support for the execution on Grids.

• Examples of Scientific WMS: Taverna, Pegasus, Triana and Kepler.

• Requirements of next generation Scientific WMS:• Management of high degree of parallelism and distribution.

• No single point of failure.

• Scalability.• Dynamicity.

2

Intr

od

uct

ion

Objectives

3

• Ensure a workflow execution:• Decentralized.• Loosely coupled (coordination mechanism).• Dynamic.• Autonomous.

“Nature-inspired metaphors have been shown to be of high interest for service coordination.”

[Viroli et al., 2009].

➔ Evaluate the viability of a nature-inspired scientific workflow system.

Intr

od

uct

ion

Chemical Programing Model (I)

4

• A program can be seen as a chemical solution:• Data: “floating” molecules in the solution.• Computation: chemical reactions between the molecules.

• Implicit parallelism and autonomy of reactions until inertia.• Expression of dynamicity.

• Data structure: Multiset (blackboard).• Containing all data molecules.• Reaction rules re-writing the multiset.

• Languages:• Gamma (Pioneered model) [Banâtre et al.,1990].• HOCL ( High-Order model) [Radenac, 2007].

Ch

emic

al P

rog

ram

ing

par

adig

m

Chemical Programing Model (II)

5

• Example:• A reaction rules is written

replace-one P by M if C

where P is a pattern which matches the required molecule, C is the reaction condition and M the result of the reaction.

Ch

emic

al P

rog

ram

ing

par

adig

m

6

HOCL-based Workflow System

Chemical Coordination: Workflow Definition

7

• Express all data and control dependencies (reaction rules and molecules).

• Molecular composition to express the logic of a workflow.

Ch

emic

al C

oo

rdin

ati o

n M

od

el

MULTISET

Chemical Coordination: Generic Rules

8

• Independent from any chemical workflow representation.• Used by chemical engines.

• Common tasks during a workflow execution:• Service invocation rule.

• Control and data transfer rule.Ch

emic

al C

oo

rdin

ati o

n M

od

el

Chemical Coordination: Workflow Patterns

9

• Control flow can be expressed using some generic rules.

• Molecular composition of composed generic rules, reactions triggering reactions.

• More patterns: parallel split, synchronization, exclusive choice, synchronization merge, cancel activity or simple merge.

Ch

emic

al C

oo

rdin

ati o

n M

od

el

Discriminator pattern

10

Architectures

• Coordination mechanism built upon HOCL.

• Two possible architectures for our workflow system:• Centralized.• Decentralized.

Centralized Architecture

11

• Central node coordinates all data and control flow between the Web services.• A chemical encapsulation per Web service participating in the workflow.• Multiset as storage space containing the workflow definition.• Chemical engine processing the content of the multiset.

Ch

emic

al W

or k

flo

w S

yste

m

Decentralized Architecture (I)

12

• Nodes communicating through a shared address space.• Persistent.• Fault-tolerant.

• Workflow executed in parts corresponding with each Web service.• Data and control transfer through this shared space.• Each node is co-responsible of the execution.

Ch

emic

al W

or k

flo

w S

yste

m

Decentralized Architecture (III)

13

• Multiset, dynamic and decentralized coordination mechanism.• Acts as a shared address space containing both control and data flows.• ChWSes communicate through the multiset. (reading and writing)• Physically distributed over ChWSes storage spaces.

Ch

emic

al W

or k

flo

w S

yste

m

14

Implementation

Centralized Prototype

15

• Service caller • Interface with all the concrete Wses.• Implemented based on Daios framework.

• HOCL Interpreter • Central engine.

• Multiset • Workflow definition.• Processed by the HOCL Interpreter.

Ch

emic

al W

or k

flo

w S

yste

m

Decentralized Prototype

16

• Chemical Web Services (ChWS):

• Service caller Interface with one concrete WS.

• Local Multiset Temporary store space.

• HOCL Interpreter Local workflow engine.

• JMS publisher/subscriber Communication module with the Multiset.

• Multiset:

• Storage space containing the whole workflow.

• Similarities with tuplespaces.

• JMS publisher/subscriber Communication module with the ChWSes.

Ch

emic

al W

or k

flo

w S

yste

m

17

Experiments

Experiments (I)

18

• Objective: Establish the viability of our chemical workflow engine in comparison with four WMS.

• Four workflow engines:

• Kepler 2.0.

• Taverna Workbench 2.2.0.

• Centralized prototype (HOCL Cen.).

• Decentralized prototype (HOCL Dec.).

• Real scenarios:

• Cardiovascular image analysis workflow (CardiacAnalysis) [7].

• Astronomical image mosaics workflow (Montage) [8].

• Bio-informatics workflow (BlastReport) [9].

• Experiments conducted on the French research infrastructure Grid'5000.

Per

form

ance

Res

ult

s

CardiacAnalysis Montage BlastReport

Num. services 6 27 5

Data exchanged High Low Medium

Coord. Complex High Medium Low

Experiments (II)

Per

form

ance

Res

ult

sP

erfo

rman

ce R

esu

lts

Experiments (II)

Per

form

ance

Res

ult

sP

erfo

rman

ce R

esu

lts

Results

21

Per

form

ance

Res

ult

s

Centralized Experiment

22

Data and computation intensive workflows.• Size and processing time increment.

Centralized coordination better for workflows with reduced computation.

Per

form

ance

Res

ult

s

Decentralized Experiment

23

Reduced computation workflows• Slightly increment of time (network latency).

Data and computation-intensive workflows show the benefits of a decentralized coordination.

Per

form

ance

Res

ult

s

Conclusion

24

• Chemical model is well featured for decentralized workflow execution. Proof of concept of the chemical workflow system.

• Our proposal: High-level decentralized coordination mechanism.

• Decentralized Architecture: Chemical web services working as local engines. Multiset as shared communication space. A High-order chemical language for workflows.

• Concepts for decentralized coordination.• Control and data driven.

Su

mm

a ry

On-going Work

25

• Implementation of a distributed multiset.

• Workflow scheduling in Federated Clouds using the chemical model.

• Modelling Agile Service Networks using the chemical choreography coordination model.

Questions ?

26

THANKS !

27