Dryad: Distributed Data-Parallel Programs from Sequential ...

Motivation and Goal Dryad Some others An improvement: Ciel Comparison Conclusion

Dryad: Distributed Data-Parallel Programs fromSequential Building Blocks

Course: CS655

Rabiet Louis

Colorado State University

Thursday 19 September 2013

1 / 48


1 Motivation and GoalWhy use Dryad ?

2 DryadGraphExampleProperties

3 Some othersAntsPegasus

4 An improvement: CielDynamic tasks

5 Comparison

6 Conclusion

2 / 48


Plan1 Motivation and Goal

Why use Dryad ?




5 Comparison

6 Conclusion

3 / 48


Why use Dryad ?

Parallel Programming

GoalMake parallel programs easier for developers.

Current Choices

Google’s MapReduce is a popular framework for developingcloud-scale applications.

GPU shader languages.

Parallel databases.

4 / 48


Why use Dryad ?



Current Choices



Parallel databases.

4 / 48


Why use Dryad ?



Current Choices



Parallel databases.

4 / 48


Why use Dryad ?



Current Choices



Parallel databases.

4 / 48


Why use Dryad ?

Solution and Results

How esle can we achieve that ?Using graphs for describing the scheduling/ relation between jobs.

The system handles the rest i.e. managing execution of stages thatcompose the graph.

ResultsCan be performance efficient and simpler.

5 / 48


Why use Dryad ?


How esle can we achieve that ?Using graphs for describing the scheduling/ relation between jobs.The system handles the rest i.e. managing execution of stages thatcompose the graph.


5 / 48


Why use Dryad ?


How esle can we achieve that ?Using graphs for describing the scheduling/ relation between jobs.The system handles the rest i.e. managing execution of stages thatcompose the graph.


5 / 48



Why use Dryad ?




5 Comparison

6 Conclusion

6 / 48


System overview

7 / 48


Graph

Dryad program is a graph

Several operators over them can be specified

Adding and cloning vertices

Adding edges

Merging two graphs

8 / 48


Graph




Adding edges

Merging two graphs

8 / 48


Graph




Adding edges

Merging two graphs

8 / 48


Graph




Adding edges

Merging two graphs

8 / 48


Graph




Adding edges

Merging two graphs

8 / 48


Graph




Adding edges

Merging two graphs

8 / 48


Graph




Adding edges

Merging two graphs

8 / 48


Graph

Merge

Merge with bypass

Asymmetric join

9 / 48


Graph

Merge

Merge with bypass

Asymmetric join

9 / 48


Graph

Merge

Merge with bypass

Asymmetric join

9 / 48


Graph

Merge

Merge with bypass

Asymmetric join

9 / 48


Graph

Merge

Merge with bypass

Asymmetric join

9 / 48


Graph

Merge

Merge with bypass

Asymmetric join

9 / 48


Example

Code example

select distinct p.objID

from photoObjAll p

join neighbors n -- call this join \X"

on p.objID = n.objID

and n.objID < n.neighborObjID

and p.mode = 1

join photoObjAll l -- call this join \Y"

on l.objid = n.neighborObjID

and l.mode = 1

and abs((p.u-p.g)-(l.u-l.g))<0.05

and abs((p.g-p.r)-(l.g-l.r))<0.05

and abs((p.r-p.i)-(l.r-l.i))<0.05

and abs((p.i-p.z)-(l.i-l.z))<0.05

10 / 48


Example

Corresponding graph

11 / 48


Example

Corresponding vertex program

GraphBu i l d e r XSet = moduleXˆN;GraphBu i l d e r DSet = moduleDˆN;GraphBu i l d e r MSet = moduleMˆ(N∗4) ;G raphBu i l d e r SSet = moduleS ˆ(N∗4) ;G raphBu i l d e r YSet = moduleYˆN;GraphBu i l d e r HSet = moduleH ˆ1 ;

G raphBu i l d e r XInputs = ( u g r i z 1 >= XSet ) | | ( n e i ghbo r >= XSet ) ;G raphBu i l d e r YInputs = ug r i z 2 >= YSet ;G raphBu i l d e r XToY = XSet >= DSet >> MSet >= SSet ;f o r ( i = 0 ; i < N∗4 ; ++i ){

XToY = XToY | | ( SSet . GetVer tex ( i ) >= YSet . GetVer tex ( i /4) ) ;}GraphBu i l d e r YToH = YSet >= HSet ;G raphBu i l d e r HOutputs = HSet >= output ;G raphBu i l d e r f i n a l = XInputs | | YInputs | | XToY | | YToH | | HOutputs ;

12 / 48


Properties

Some nice properties ...

ChannelsCommunication can be chosen between

Files

TCP Pipe

Shared-Memory FIFO

Support for legacy applications

Node/vertices can execute old code not specifically written forDryad.

13 / 48


Properties

Some nice properties ...

ChannelsCommunication can be chosen between

Files

TCP Pipe

Shared-Memory FIFO

Support for legacy applications

Node/vertices can execute old code not specifically written forDryad.

13 / 48


Properties

Some nice properties

Runtime aggregation

Fault toleranceRestart failed vertex.

14 / 48


Properties

Some nice properties

Runtime aggregation

Fault toleranceRestart failed vertex.

14 / 48


Properties

Scalability ...

15 / 48


Properties

Scalability

16 / 48


Properties

SQL

17 / 48



Why use Dryad ?




5 Comparison

6 Conclusion

18 / 48


Ants

Some limitations of Dryad

7 Fault tolerance: what happens if a stage manager fails.

7 No way of defining dynamic graphs.

19 / 48


Ants




19 / 48


Ants




19 / 48


Ants

Linking ACO and Dryad

Finding a path between producers and consumers

Is finding a path sufficient ?

What happens if a node is already close to overutilization ?

1 Need a way of evaluating the quality of the path.

2 Avoid interference between two tasks.

20 / 48


Ants







20 / 48


Ants







20 / 48


Ants







20 / 48


Ants







20 / 48


Ants







20 / 48


Ants







20 / 48


Ants

Abstraction of the problem

Each cell is a server

In a cell C, we have

N(C ), the neighboursΦC

D,n: probability if the ant has the destination D that the antis going to the cell n when currently on C .Routing table: mean and variance of the time to go from C toD.

21 / 48


Ants






21 / 48


Ants






21 / 48


Ants






21 / 48


Ants

First type of ant: routing ants

An ant records its path, knows the final destination D and thesource Cs . At each step, when an ant is on a cell C (different fromD).

1 Record queue waiting time at C .

2 Choose next hop (unknown neighbour first) using ΦCD,n.

[Detection of cycles]

3 En-queue on the link to go to n.

4 Record waiting time and transmission time.

5 If the ant has not arrived at D, insert itself in the queue,repeat 1-5.

6 If the ant arrived at D, start to go back to Cs

22 / 48


Ants










22 / 48


Ants










22 / 48


Ants










22 / 48


Ants










22 / 48


Ants










22 / 48


Ants










22 / 48


Ants

Second type: scouting ants

Try to find alternative path to go from Cs to D.

If the computing time of the new tasks Lnew and thecomputing time of the old tasks Lprev does not surpass athreshold LT (user defined).

if LT is not defined, the ant just ensures that the load on theserver is inferior to α (random variable). g is an other randomvariable to determine the number of tasks the ant will placeon the server.

The system generate scout ants with various α and g . If someants are not able to place all the tasks and have reached Dthen they will crawl back and modify the ΦC

D,n (NegativeReinforcement).

23 / 48


Ants







23 / 48


Ants







23 / 48


Ants







23 / 48


Ants

Third type: enforcement ants

When enough scout ants are reporting success, someenforcement ants are released.

Except if extraordinary results are shown by scouts reports,the number of successful scout ants need to be higher than athreshold.

The enforcement ants recompute the cost (and compare itwith the scout results); if everything is fine the placement isdone.

24 / 48


Ants





24 / 48


Ants





24 / 48


Ants





24 / 48


Ants

Some drawbacks

7 Evaluation is done via simulation

7 No theoretical proof of correctness for placement algorithm

7 Very few optimizations → what about overhead costs ?

25 / 48


Ants

Some drawbacks




25 / 48


Ants

Some drawbacks




25 / 48


Ants

Some drawbacks




25 / 48


Pegasus

26 / 48


Pegasus

27 / 48


Pegasus

Scheduling horizon

28 / 48



Why use Dryad ?




5 Comparison

6 Conclusion

29 / 48


Dynamic tasks

Control Flow

30 / 48


Dynamic tasks

Publishing: how to handle dynamic graphs

31 / 48


Dynamic tasks

Lazy evaluation

Lazy evaluation is used to schedule task.

1 Start by the (expected) output of the root task

2 If T has only concrete dependencies, execute it immediately

3 Else block T until the input becomes concrete.

32 / 48


Dynamic tasks

Lazy evaluation





32 / 48


Dynamic tasks

Lazy evaluation





32 / 48


Dynamic tasks

Lazy evaluation





32 / 48


Dynamic tasks

An optimization

result = spawn_exec(executor, args, n);

Unique Naming

for the i th output: name = executor:SHA1(arg‖n):i

MemoisationIf a new task’s outputs have already been produced by a previoustask, the new task need not be executed at all.

33 / 48


Master

Ciel support master failure

34 / 48


Persistent logging

Write when a new job is launched

The descriptors (graphs) are periodically written to the disk.

If the master reboots, it looks for unmatched jobs. They arethen restarted (the graph has been stored).

35 / 48


Persistent logging




35 / 48


Persistent logging




35 / 48


Secondary masters

Everything is sent to every master (primary and secondaries).

Every secondary records workers address.

36 / 48


Secondary masters



36 / 48


Secondary masters



36 / 48


Object table reconstruction

In a case of a master rebooting (failure then recovering):

1 Workers detect master failure (heartbeat).

2 If a failure is detected, workers send registration to the master.

3 Master resends objects list.

37 / 48







37 / 48







37 / 48







37 / 48


Speed comparison

38 / 48



Why use Dryad ?




5 Comparison

6 Conclusion

39 / 48


Map Reduce

3 Simple

3 Efficient for a specific kind of programs: web indexing

7 Reliability very simple

7 Workflow hardcoded

7 New ? PL-SQL, Teradata.

Improvement

Instead of having a Map phase then a Reduce phase: do acombination of Map-Reduce.

Computation Reuse !

40 / 48


Map Reduce

3 Simple





Improvement


Computation Reuse !

40 / 48


Map Reduce

3 Simple





Improvement


Computation Reuse !

40 / 48


Map Reduce

3 Simple





Improvement


Computation Reuse !

40 / 48


Map Reduce

3 Simple




7 New ?

PL-SQL, Teradata.

Improvement


Computation Reuse !

40 / 48


Map Reduce

3 Simple





Improvement


Computation Reuse !

40 / 48


Map Reduce

3 Simple





Improvement


Computation Reuse !

40 / 48


Map Reduce

3 Simple





Improvement


Computation Reuse !

40 / 48


Dryad

3 Some optimizations: aggregation (data locality).

3 Much more general than Map Reduce

7 Not so simple (look at the code)

7 No cycles

7 Coming from SQL.

Improvement

Ciel

41 / 48


Dryad




7 No cycles

7 Coming from SQL.

Improvement

Ciel

41 / 48


Dryad




7 No cycles

7 Coming from SQL.

Improvement

Ciel

41 / 48


Dryad




7 No cycles

7 Coming from SQL.

Improvement

Ciel

41 / 48


Dryad




7 No cycles

7 Coming from SQL.

Improvement

Ciel

41 / 48


Dryad




7 No cycles

7 Coming from SQL.

Improvement

Ciel

41 / 48


Dryad




7 No cycles

7 Coming from SQL.

Improvement

Ciel

41 / 48


Ants

3 Analogy with nature.

3 Distributed

3 General

3 Account for machine load

7 Difficult to predict behaviour

7 Overhead !!

Improvement

What happens if an ant is faulty ?

I would replace it with a simpler search of optimal paths.

42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Ants


3 Distributed

3 General



7 Overhead !!

Improvement



42 / 48


Pegasus

3 Performance issue

7 Not general enough

Improvement

Reliability (only partition level)

43 / 48


Pegasus

3 Performance issue


Improvement


43 / 48


Pegasus

3 Performance issue


Improvement


43 / 48


Pegasus

3 Performance issue


Improvement


43 / 48


Pegasus

3 Performance issue


Improvement


43 / 48


Ciel

3 Much more general.

3 Dynamic task dependencies

Improvement

Random funtions

How far are you from machine performance ?

44 / 48


Ciel



Improvement

Random funtions


44 / 48


Ciel



Improvement

Random funtions


44 / 48


Ciel



Improvement

Random funtions


44 / 48


Ciel



Improvement

Random funtions


44 / 48


Ciel



Improvement

Random funtions


44 / 48


Perpectives

Adding user’s transformations for graphs: tiling, merge, etc...

BGP (distance-vector routing protocol)

Look at this problem using maximum flow problem.

45 / 48


Perpectives




45 / 48


Perpectives




45 / 48


Perpectives




45 / 48



Why use Dryad ?




5 Comparison

6 Conclusion

46 / 48


Conclusion

Microsoft decided to stop developing Dryad for commercialreasons.

47 / 48


Conclusion

Summary

Several restrictions that make the result possible:

Node results are deterministic

Job manager doesn’t fail

Acyclic graph

Doesn’t add enough to MapReduce

48 / 48

Dryad: Distributed Data-Parallel Programs from Sequential ...

Documents