Designing and Evaluating a Distributed Computing Language ... · Evaluating a Distributed Computing Language Runtime Christopher Meiklejohn ... 26 %% Create initial ... Selective

Designing and Evaluating a Distributed Computing Language RuntimeChristopher Meiklejohn (@cmeik) Université catholique de Louvain, Belgium

1

RA

RB

RA

RB

1

set(1)

RA

RB

1

set(1)

3

2

set(2)

set(3)

RA

RB

1

set(1)

3

2

set(2)

set(3)

?

?

Synchronization• To enforce an order

Makes programming easier

6



• Eliminate accidental nondeterminismPrevent race conditions

6



• Eliminate accidental nondeterminismPrevent race conditions

• TechniquesLocks, mutexes, semaphores, monitors, etc.

6

Difficult Cases• “Internet of Things”,

Low power, limited memory and connectivity

7

Difficult Cases• “Internet of Things”,

Low power, limited memory and connectivity

• Mobile GamingOffline operation with replicated, shared state

7

Weak Synchronization• Can we achieve anything without synchronization?

Not really.

8


Not really.

• Strong Eventual Consistency (SEC)“Replicas that deliver the same updates have equivalent state”

8


Not really.


• Primary requirementEventual replica-to-replica communication

8


Not really.



• Order insensitive! (Commutativity)

8


Not really.



• Order insensitive! (Commutativity)

• Duplicate insensitive! (Idempotent)

8

RA

RB

RA

RB

1

set(1)

RA

RB

1

set(1)

3

2

set(2)

set(3)

RA

RB

1

3

2

3

3

set(1) set(2)

set(3)

max(2,3)

max(2,3)

How can we succeed with Strong Eventual Consistency?

13

Programming SEC1. Eliminate accidental nondeterminism

(ex. deterministic, modeling non-monotonic operations monotonically)

14



2. Retain the properties of functional programming(ex. confluence, referential transparency over composition)

14




3. Distributed, and fault-tolerant runtime(ex. replication, membership, dissemination)

14





15

Convergent ObjectsConflict-Free Replicated Data Types

16SSS 2011

Conflict-Free Replicated Data Types

• Many types exist with different propertiesSets, counters, registers, flags, maps, graphs

17

Conflict-Free Replicated Data Types

• Many types exist with different propertiesSets, counters, registers, flags, maps, graphs

• Strong Eventual ConsistencyInstances satisfy SEC property per-object

17

RA

RB

RC

RA

RB

RC

{1}

(1, {a}, {})

add(1)

RA

RB

RC

{1}

(1, {a}, {})

add(1)

{1}

(1, {c}, {})

add(1)

RA

RB

RC

{1}

(1, {a}, {})

add(1)

{1}

(1, {c}, {})

add(1)

{}

(1, {c}, {c})

remove(1)

RA

RB

RC

{1}

(1, {a}, {})

add(1)

{1}

(1, {c}, {})

add(1)

{}

(1, {c}, {c})

remove(1)

{1}

{1}

{1}

(1, {a, c}, {c})

(1, {a, c}, {c})

(1, {a, c}, {c})





23

Convergent Programs Lattice Processing

24PPDP 2015

Lattice Processing (Lasp)• Distributed dataflow

Declarative, functional programming model

25



• Convergent data structuresPrimary data abstraction is the CRDT

25



• Convergent data structuresPrimary data abstraction is the CRDT

• Enables compositionProvides functional composition of CRDTs that preserves the SEC property

25

26

%% Create initial set. S1 = declare(set),

%% Add elements to initial set and update. update(S1, {add, [1,2,3]}),

%% Create second set. S2 = declare(set),

%% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

27





28





29





30









31

Distributed Runtime Selective Hearing

32W-PSDS 2015

Selective Hearing• Epidemic broadcast based runtime system

Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution

33



• Well-matched to Lattice Processing (Lasp)

33




• Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient

33





• Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures

33





• Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures

• “Selective Receive”Nodes selectively receive and process messages based on interest.

33

Layered Approach

34

Layered Approach• Membership

Configurable membership protocol which can operate in a client-server or peer-to-peer mode

34



• Broadcast (via Gossip, Tree, etc.)Efficient dissemination of both program state and application state via gossip, broadcast tree, or hybrid mode

34



• Broadcast (via Gossip, Tree, etc.)Efficient dissemination of both program state and application state via gossip, broadcast tree, or hybrid mode

• Auto-discoveryIntegration with Mesos, auto-discovery of Lasp nodes for ease of configurability

34

Membership Overlay

Membership Overlay

Broadcast Overlay

Membership Overlay

Broadcast Overlay

Membership Overlay

Broadcast Overlay

Mobile Phone

Membership Overlay

Broadcast Overlay

Mobile Phone

Distributed Hash Table

Membership Overlay

Broadcast Overlay

Mobile Phone

Distributed Hash Table

Lasp Execution





41

What can we build? Advertisement Counter

42

Advertisement Counter• Mobile game platform selling

advertisement spaceAdvertisements are paid according to a minimum number of impressions

43

Advertisement Counter• Mobile game platform selling

advertisement spaceAdvertisements are paid according to a minimum number of impressions

• Clients will go offlineClients have limited connectivity and the system still needs to make progress while clients are offline

43

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation

User-Maintained CRDT

Lasp-Maintained CRDT

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Client Side, Single Copy at Client

44

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


45

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


46

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


47

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


48

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


49

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


50

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


51

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


52

Ads

Rovio Ad Counter 1

Rovio Ad Counter 2

Riot Ad Counter 1

Riot Ad Counter 2

Contracts

AdsContracts

AdsWith

Contracts

Riot Ads

Rovio Ads

FilterProduct

Read � 50,000

Remove

Increment

Read

Union

Lasp Operation



Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1

Rovio Ad Counter

1

Rovio Ad Counter

2

Riot Ad Counter

1


Evaluation Initial Evaluation

53

Background Distributed Erlang

• Transparent distributionBuilt-in, provided by Erlang/BEAM, cross-node message passing.

54



• Known scalability limitationsAnalyzed in academic in various publications.

54




• Single connectionHead of line blocking.

54




• Single connectionHead of line blocking.

• Full membershipAll-to-all failure detection with heartbeats and timeouts.

54

Background Erlang Port Mapper Daemon• Operates on a known port

Similar to Solaris sunrpc style portmap: known port for mapping to dynamic port-based services.

55

Background Erlang Port Mapper Daemon• Operates on a known port

Similar to Solaris sunrpc style portmap: known port for mapping to dynamic port-based services.

• Bridged networkingProblematic for cluster in bridged networking with dynamic port allocation.

55

Experiment Design• Single application

Advertisement counter example from Rovio Entertainment.

56



• Runtime configurationApplication controlled through runtime environment variables.

56




• MembershipFull membership with Distributed Erlang via EPMD.

56




• MembershipFull membership with Distributed Erlang via EPMD.

• DisseminationState-based object dissemination through anti-entropy protocol (fanout-based, PARC-style.)

56

Experiment Orchestration• Docker and Mesos with Marathon

Used for deployment of both EPMD and Lasp application.

57



• Single EPMD instance per slaveControlled through the use of host networking and HOSTNAME: UNIQUE constraints in Mesos.

57




• LaspLocal execution using host networking: connects to local EPMD.

57




• LaspLocal execution using host networking: connects to local EPMD.

• Service DiscoveryService discovery facilitated through clustering EPMD instances through Sprinter.

57

Ideal Experiment• Local Deployment

High thread concurrency when operating with lower node count.

58

Ideal Experiment• Local Deployment

High thread concurrency when operating with lower node count.

• Cloud DeploymentLow thread concurrency when operating with a higher node count.

58

Results Initial Evaluation

59

Initial Evaluation• Moved to DC/OS exclusively

Environments too different: too much work needed to be adapted for things to work correctly.

60



• Single orchestration taskDispatched events, controlled when to start and stop the evaluation and performed log aggregation.

60




• BottleneckEvents immediately dispatched: would require blocking for processing acknowledgment.

60




• BottleneckEvents immediately dispatched: would require blocking for processing acknowledgment.

• UnrealisticEvents do not queue up all at once for processing by the client.

60

Lasp Difficulties• Too expensive

2.0 CPU and 2048 MiB of memory.

61



• Weeks spent adding instrumentationProcess level, VM level, Erlang Observer instrumentation to identify heavy CPU and memory processes.

61




• Dissemination too expensive1000 threads to a single dissemination process (one Mesos task) leads to backed up message queues and memory leaks.

61




• Dissemination too expensive1000 threads to a single dissemination process (one Mesos task) leads to backed up message queues and memory leaks.

• UnrealisticTwo different dissemination mechanisms: thread to thread and node to node: one is synthetic.

61

EPMD Difficulties• Nodes become unregistered

Nodes randomly unregistered with EPMD during execution.

62



• Lost connectionEPMD loses connections with nodes for some arbitrary reason.

62



• Lost connectionEPMD loses connections with nodes for some arbitrary reason.

• EPMD task restarted by MesosRestarted for an unknown reason, which leads Lasp instances to restart in their own container.

62

Overhead Difficulties• Too much state

Client would ship around 5 GiB of state within 90 seconds.

63



• Delta disseminationDelta dissemination only provides around a 30% decrease in state transmission.

63



• Delta disseminationDelta dissemination only provides around a 30% decrease in state transmission.

• Unbounded queuesMessage buffers would lead to VMs crashing because of large memory consumption.

63

Evaluation Rearchitecture

64

Ditch Distributed Erlang• Pluggable membership service

Build pluggable membership service with abstract interface initially on EPMD and later migrate after tested.

65



• Adapt Lasp and Broadcast layerIntegrate pluggable membership service throughout the stack and librate existing libraries from distributed Erlang.

65



• Adapt Lasp and Broadcast layerIntegrate pluggable membership service throughout the stack and librate existing libraries from distributed Erlang.

• Build service discovery mechanismMechanize node discovery outside of EPMD based on new membership service.

65

Partisan (Membership Layer)

• Pluggable protocol membership layerAllow runtime configuration of protocols used for cluster membership.

66



• Several protocol implementations:

66




• Full membership via EPMD.

66





• Full membership via TCP.

66






• Client-server membership via TCP.

66







• Peer-to-peer membership via TCP (with HyParView)

66







• Peer-to-peer membership via TCP (with HyParView)

• VisualizationProvide a force-directed graph-based visualization engine for cluster debugging in real-time.

66

Partisan (Full via EPMD or TCP)

• Full membershipNodes have full visibility into the entire graph.

67



• Failure detectionPerformed by peer-to-peer heartbeat messages with a timeout.

67




• Limited scalabilityHeartbeat interval increases when node count increases leading to false or delayed detection.

67




• Limited scalabilityHeartbeat interval increases when node count increases leading to false or delayed detection.

• TestingUsed to create the initial test suite for Partisan.

67

Partisan (Client-Server Model)

• Client-server membershipServer has all peers in the system as peers; client has only the server as a peer.

68



• Failure detectionNodes heartbeat with timeout all peers they are aware of.

68




• Limited scalabilitySingle point of failure: server; with limited scalability on visibility.

68




• Limited scalabilitySingle point of failure: server; with limited scalability on visibility.

• TestingUsed for baseline evaluations as “reference” architecture.

68

Partisan (HyParView, default)

• Partial view protocolTwo views: active (fixed) and passive (log n); passive used for failure replacement with active view.

69



• Failure detectionPerformed by monitoring active TCP connections to peers with keep-alive enabled.

69



• Failure detectionPerformed by monitoring active TCP connections to peers with keep-alive enabled.

• Very scalable (10k+ nodes during academic evaluation)However, probabilistic; potentially leads to isolated nodes during churn.

69

Sprinter (Service Discovery)

• Responsible for clustering tasksUses Partisan to cluster all nodes and ensure connected overlay network: reads information from Marathon.

70



• Node localOperates at each node and is responsible for taking actions to ensure connected graph: required for probabilistic protocols.

70



• Node localOperates at each node and is responsible for taking actions to ensure connected graph: required for probabilistic protocols.

• Membership mode specificKnows, based on the membership mode, how to properly cluster nodes and enforces proper join behaviour.

70

Debugging Sprinter• S3 archival

Nodes periodically snapshot their membership view for analysis.

71



• Elected node (or group) analyses Periodically analyses the information in S3 for the following:

71




• Isolated node detection Identifies isolated nodes and takes corrective measures to repair the overlay.

71





• Verifies symmetric relationship Ensures that if a node knows about another node, the relationship is symmetric: prevents I know you, but you don’t know me.

71





• Verifies symmetric relationship Ensures that if a node knows about another node, the relationship is symmetric: prevents I know you, but you don’t know me.

• Periodic alertingAlerts regarding disconnected graphs so external measures can be taken, if necessary.

71

Evaluation Next Evaluation

72

Evaluation Strategy• Deployment and runtime configuration

Ability to deploy a cluster of node and configure simulations at runtime.

73



• Each simulation:

73




• Different application scenario Uniquely execute a different application scenario at runtime based on runtime configuration.

73





• Result aggregation Aggregate results at end of execution and archive these results.

73






• Plot generationAutomatically generate plots for the execution and aggregate the results of multiple executions.

73






• Plot generationAutomatically generate plots for the execution and aggregate the results of multiple executions.

• Minimal coordination Work must be performed with minimal coordination, as a single orchestrator is a scalability bottleneck for large applications.

73

Completion Detection• “Convergence Structure”

Uninstrumented CRDT of grow-only sets containing counters that each node manipulates.

74



• Simulates a workflowNodes use this operation to simulate a lock-stop workflow for the experiment.

74




• Event GenerationEvent generation toggles a boolean for the node to show completion.

74





• Log Aggregation Completion triggers log aggregation.

74






• ShutdownUpon log aggregation completion, nodes shutdown.

74






• ShutdownUpon log aggregation completion, nodes shutdown.

• External monitoringWhen events complete execution, nodes automatically begin the next experiment.

74

Results Next Evaluation

75

Results Lasp• Single node orchestration: bad

Not possible once you exceed a few nodes: message queues, memory, delays.

76



• Partial ViewsRequired: rely on transitive dissemination of information and partial network knowledge.

76



• Partial ViewsRequired: rely on transitive dissemination of information and partial network knowledge.

• ResultsReduced Lasp memory footprint to 75MB; larger in practice for debugging.

76

Results Partisan• Fast churn isolates nodes

Need a repair mechanism: random promotion of isolated nodes; mainly issues of symmetry.

77



• FIFO across connectionsNot per connection, but protocol assumes across all connections leading to false disconnects.

77




• Unrealistic system modelYou need per message acknowledgements for safety.

77




• Unrealistic system modelYou need per message acknowledgements for safety.

• Pluggable protocol helps debuggingBeing able to switch to full membership or client-server assists in debugging protocol vs. application problems.

77

Latest Results• Reproducibility at 300 nodes for full applications

Connectivity, but transient partitions and isolated nodes at 500 - 1000 nodes (across 140 instances.)

78



• Limited financially and by AmazonHarder to run larger evaluations because we’re limited financially (as a university) and because of Amazon limits.

78



• Limited financially and by AmazonHarder to run larger evaluations because we’re limited financially (as a university) and because of Amazon limits.

• Mean state reduction per clientAround 100x improvement from our PaPoC 2016 initial evaluation results.

78

Plat à emporter• Visualizations are important!

Graph performance, visualize your cluster: all of these things lead to easier debugging.

79



• Control changesNo Lasp PR accepted without divergence, state transmission, and overhead graphs.

79




• AutomationDevelopers use graphs when they are easy to make: lower the difficulty for generation and understand how changes alter system behaviour.

79




• AutomationDevelopers use graphs when they are easy to make: lower the difficulty for generation and understand how changes alter system behaviour.

• Make work easily testableWhen you test locally and deploy globally, you need to make things easy to test, deploy and evaluate (for good science, I say!)

79

80

Christopher Meiklejohn@cmeik http://www.lasp-lang.org http://github.com/lasp-lang

Thanks!

http://www.lasp-lang.org

http://github.com/lasp-lang

Designing and Evaluating a Distributed Computing Language ... · Evaluating a Distributed Computing Language Runtime Christopher Meiklejohn ... 26 %% Create initial ... Selective

Documents