Large Partially-connected Erlang Clusters · Large Partially-connected Erlang Clusters Motiejus Jak stys @mo kelione Sr. backend developer Erlang User Conference 2014 Stockholm. 2/34

1/34

Large Partially-connected ErlangClusters

Motiejus Jakstys @mo kelione

Sr. backend developer

Erlang User Conference 2014Stockholm

https://twitter.com/mo_kelione

2/34

Background and Agenda

Big service-oriented backend infrastructure.

I Infrastructure which provides API for ourgaming portals

I A couple of dozen servers running about 50services each

Agenda:

I Historical introduction

I Technical stuff

Don’t hesitate to interrupt!

2/34





Agenda:


I Technical stuff


2/34





Agenda:


I Technical stuff


3/34

My point today

Management of big battles is very similar to runningdistributed systems.

I have a good example for you.

3/34

My point today



3/34

My point today



4/34

Battle of Stalingrad1942.08.23 – 1943.02.02

Image source: Deutsches Bundesarchiv, RIA Novosti Archive

5/34

Rubble battle

Image source: Deutsches Bundesarchiv, RIA Novosti Archive

6/34

Peer-to-peer communication

I Impossible to determine location.

I Radio was unreliable and useless.

I Reinforcement/supply requests were just voice.

7/34

Heterogeneous

I Infantry

I Tank fleet

I Air Force

I Medical staff

I Commanders

I ...

Image source: http://navy.mil

http://navy.mil

8/34

Communication channelsIt is clear who gets ordersfrom who. Very clear.

Image source:http://www.vetfriends.com/military_structure/

http://www.vetfriends.com/military_structure/

9/34

Dynamic environment

Nature of the battle is dynamic:

Losses and reinforcements change the dynamics ofthe battlefield.

9/34

Dynamic environment

Nature of the battle is dynamic:Losses and reinforcements change the dynamics ofthe battlefield.

10/34

Big Plan

It is not enough to only take care of your business.All units must work to achieve a common goal.

11/34

Situation at 1942.11.15

Dire Soviet situation:

I Huge causalities

I Red Army lifeexpectancy:

I Soldier: < 1 dayI Officer: < 3 days

I Germans have 90%of the city

Image source: Antill, P., Dennis, P. Stalingrad 1942(Campaign). Osprey Publishing (June 19, 2007).

11/34



I Huge causalitiesI Red Army life

expectancy:I Soldier: < 1 dayI Officer: < 3 days



11/34



I Huge causalitiesI Red Army life

expectancy:I Soldier: < 1 dayI Officer: < 3 days



12/34

Situation after 1942.11.23

I Germans surroundedby soviets

I City is Germans’,Germans’ areSoviets’

I Turning point of thebattle

I 6’th Army (thesurrounded one) wasdestroyed.


12/34







12/34







12/34







13/34

Outline

1 Historical introduction

2 Technical stuffMotivationFeaturesAPI

3 QA

You sure you have no questions?

14/34

Set the groundsWe have a lot of Erlang nodes trying to achieve acommon goal.

I ≈ 50 nodes per server

I a couple of dozens of servers

Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...

How to connect them?

14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


14/34




Example services:

I Higscores

I Authentication

I Chat

I User profiles

I ...


15/34

Peer-to-peer communication

You don’t want bottlenecks.You don’t want single points of failure.

16/34

Dynamic nodes

Nodes and services start and stop all the time.

The system must continue to function and self-heal.

16/34

Dynamic nodes

Nodes and services start and stop all the time.The system must continue to function and self-heal.

17/34

Partially connected networkn : number of nodes.Total connections = n(n−1)

2

0

2 M

4 M

6 M

8 M

10 M

12 M

14 M

0 20 40 60 80 100

Total connections, 50 nodes per machine

17/34

Partially connected networkn : number of nodes.Total connections = n(n−1)

2

0

2 M

4 M

6 M

8 M

10 M

12 M

14 M

0 20 40 60 80 100

Total connections, 50 nodes per machine

18/34

Conclusions (for now)

It pays off to optimize the topology so

communication is more effective, e.g.:

I Army

I Software Defined Networking

I Management

19/34

Remember?

Erlang BattleP2P communication P2P communicationMulti app groups HeterogeneousDynamic nodes Dynamic environmentPartially connected network Communication channels

Maintaining a distributed system is like managing a battle.

We can be generals.

19/34

Remember?



We can be generals.

19/34

Remember?



We can be generals.

20/34

We need to solve this

What spapi-router pg2 gprocP2P communication X X XMulti app groups X X XDynamic nodes X X XPartially connected networkby limiting connections

X x x

21/34

spapi-router features in a nutshell

A library.

I Creates a mesh networkConnects (hidden) nodes like configured

I Abstracts destinationRPC-call a service, not a node

I Mature and optimizedUsed for 2,5 years in a sufficiently large SOA

I Instrumented

21/34


A library.




I Instrumented

21/34


A library.




I Instrumented

21/34


A library.




I Instrumented

22/34

For example

[email protected]

kernel stdlibpagebuilderspapi_router

kernelstdlibheader

kernelstdlibmainsection


[email protected]

[email protected]

[email protected]

[email protected]


Configuration [email protected]:

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [

{"^header[0-9]*", [header]},

{"^mainsec[0-9]*", [mainsection]},

]}

]}

22/34

For example

[email protected]

kernel stdlibpagebuilderspapi_router

kernelstdlibheader



[email protected]

[email protected]

[email protected]

[email protected]


Configuration [email protected]:

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [



]}

]}

23/34

More configuration

I hosts monitor interval ms

I world monitor interval ms

I worker monitor interval ms

I callback module

24/34

How it works

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [



]}

]}

1. Connects tohost names

2. Asks EPMD forrunning nodes

3. Connects to nodesmatching regexp

4. Checks forapplications innodes

5. Connects torelevant nodes

24/34

How it works

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [



]}

]}






24/34

How it works

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [



]}

]}






24/34

How it works

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [



]}

]}






24/34

How it works

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [



]}

]}






24/34

How it works

{spapi_router, [

{host_names, [

"host1.fqdn",

"host2.fqdn",

]},

{workers, [



]}

]}






25/34

Why instrument?Helps understand the system is sound.

The first thing you want to instrument is the border of yourservice.Effortless instrumentation for all calls via spapi-router.

26/34

callback module #1

%% New resource is detected

-callback new_resource({Service :: atom(), node()},

opts()) -> any().

%% Existing resource is lost

%% (node disconnect, shutdown, etc).

-callback lost_resource({Service :: atom(), node()},

opts()) -> any().

27/34

callback module #2

-type log_spec() :: {

Service :: atom(),

Module :: atom(),

Function :: atom()

}.

%% Time instrumentation

-callback measure(log_spec(),

fun(() -> A), opts()) -> A.

%% Called on success/failure of a function call.

-callback success(log_spec(), opts()) -> term().

-callback failure(log_spec(), opts()) -> term().

28/34

Calling others

I spr_router:call(piqi_rpc, erlang, node, []).

I spr_router:call_all(piqi_rpc, erlang, node, []).

Extras:

I call/5

I call all/5

I list workers/0

I list workers/1

I list hosts/0

28/34

Calling others



Extras:

I call/5

I call all/5

I list workers/0

I list workers/1

I list hosts/0

28/34

Calling others



Extras:

I call/5

I call all/5

I list workers/0

I list workers/1

I list hosts/0

28/34

Calling others



Extras:

I call/5

I call all/5

I list workers/0

I list workers/1

I list hosts/0

29/34

Future optimizations

I Takes time to figure out a ’stop’.

I Monitor application controller instead ofnode.

I One node == one service.

30/34

How to change nodes?

Puppet plus

I RelUp

I ... or anything really:spr app:config change([], [], []).

31/34

Battle stories

I Tried to disconnect from irrelevant nodes first

I what if relationship is one-way?I one misbehaving component can bring the system

down

I Fully connected network experience (> 1Knodes)

I everyone should try thatI thanks to off-peak and 10G NIC

31/34

Battle stories



down



31/34

Battle stories


I what if relationship is one-way?

I one misbehaving component can bring the systemdown



31/34

Battle stories



down



31/34

Battle stories



down



31/34

Battle stories



down


I everyone should try that

I thanks to off-peak and 10G NIC

31/34

Battle stories



down



32/34

Stats

2012-01-12 Initial commit (Thijs Terlouw)

2012-02-16 1.0.0 – first prod (Thijs Terlouw)

2012-08-24 1.2.0 – broadcast (Enrique Paz)

2014-06-20 2.1.0 – many contributions by Spil Games

2014-07-04 Public pre-release

commit b3b7aad9ca14ed230f28635826b371b6bbea3840

Author: Motiejus Jakstys <[email protected]>

Date: Wed Jun 4 14:39:40 2014 +0200

Initial commit

21 files changed, 2984 insertions(+)

2014-07-10 http://github.com/spilgames/spapi-router

http://github.com/spilgames/spapi-router

32/34

Stats








Date: Wed Jun 4 14:39:40 2014 +0200

Initial commit




32/34

Stats








Date: Wed Jun 4 14:39:40 2014 +0200

Initial commit




32/34

Stats








Date: Wed Jun 4 14:39:40 2014 +0200

Initial commit




32/34

Stats








Date: Wed Jun 4 14:39:40 2014 +0200

Initial commit




32/34

Stats








Date: Wed Jun 4 14:39:40 2014 +0200

Initial commit




33/34

Outline

1 Historical introduction

2 Technical stuffMotivationFeaturesAPI

3 QA

You sure you have no questions?

34/34

QA

Large Partially-connected Erlang Clusters · Large Partially-connected Erlang Clusters Motiejus Jak stys @mo kelione Sr. backend developer Erlang User Conference 2014 Stockholm. 2/34

Documents