CloudConf2017 - Deploy, Scale & Coordinate a microservice oriented application

CloudConfTurin 16th March 2017

Deploy, scale and coordinate a microservice orientedapplication

cloudconf.it

Walter Dal Mut

walter.dalmut @ corley.itSolution Architect @

Corley

Microservices

They're coming outta the walls

Docker Swarm

Issue a new process (deploy)$ docker service create \ --name app \ registry.walterdalmut.com/app:v1

$ ./app &

Run different processes (scale)$ docker service scale app=10 $ ./app &

$ ./app & $ ./app &

What about the locking system?In a distributed system: locks (distributed locks) are

the foundations for activity synchronization

What about coordination?

A service have its own con�guration atlaunch I am here with this address, port etc...

A service require other servicecon�gurations Where the database is, which password i should use, etc...

The service should reports its owns status I am alive and responsive (for healthcheck) other services can check the health report for the maintenance mode or to shortcircuit the service dependency

To expose the coordination problem we create anapplication

Read the twitter stream

#cloudconf2017

users that tweet with this handle create a reserved API service [ JSON over HTTP ]

GET /tweet - list my tweetsPOST /tweet - record a newtweet

The database to store tweets is self-contained in the API service

Now every service is exposed with a unique pairaddress:port in the swarm

every box is a container (service)blinks for activities (publish new tweets)

Multiple services(1)(1)

(1..*)

(n)

(1..*)

Stream readererDistributed queueService Worker (need a distributed lock)

a lock identi�es the service deploymentprogresson missing => service deployon existing => publish messages

Per user containerA proxy to list users and redirects requests

HTTP framework> GET /user/walterdalmut HTTP/1.1 > Host: cluster.corsi.walterdalmut.com:30000 > User-Agent: curl/7.47.0 > Accept: */*

< HTTP/1.1 302 Found < location: http://cluster.corsi.walterdalmut.com:30002/v1/tweet < vary: origin< cache-control: no-cache < content-length: 0< Date: Sun, 12 Mar 2017 11:26:50 GMT < Connection: keep-alive

CloudConf2017 Example

Every user have its own network address andportEvery user expose its own API

How do we connect services together?

DNS as a coordination systemDNS is a good solution to point things in a network

DNS SRV expose a service address con�guration$ dig srv _auth._tcp.walterdalmut.com +short 1 10 8080 1.api.walterdalmut.com 1 10 8080 2.api.walterdalmut.com

And service con�gurations?host: db.mynet.local port: 3306 username: root password: root dbname: example

K / VSeveral coordination systems available

Etcd is one of the most interesting coordination system available Consul integrates di�erent things together like: DNS, KV, etc...

many other: zookeeper, etc...

Redis as a coordination service

distribute con�gurations at paths$ cat mydb.conf | redis-cli set /path/to/disk/mydb.conf -

Where is my `ls` command now?$ redis-cli keys /path/* 1) "/path/to/disk/mydb.conf"

Get my con�guration back$ redis-cli get /path/to/disk/mydb.conf host: db.mynet.local port: 3306 username: root password: root dbname: example

How to report the application status?Healthchecks

Con�gurations can also expiresDead man switch

application reports continuously

cat mydb.conf | SETEX /path/to/disk/mydb.conf 30 -

EXPIRE /path/to/disk/mydb.conf 30 ... sleep 20 EXPIRE /path/to/disk/mydb.conf 30 ...

Services links togetherCan i watch for con�guration changes?

refresh my services on updates

Redis Keyspace Noti�cations

or in your con�guration �le

CONFIG SET notify-keyspace-events AKE

Listen for my con�guration changesSUBSCRIBE __keyspace@0__:/path/to/disk/mydb.conf

Here the event$ cat mydb.conf | redis-cli set /path/to/disk/mydb.conf -

1) "message" 2) "__keyspace@0__:/path/to/disk/mydb.conf" 3) "set"

Distributed locksIn a single node for redis

NX - if not exists PX 30000 - expires in 30000 ms

SET /etc/lock/username/.lock {random_value} NX PX 30000

After 30 seconds the lock expires

SET /etc/lock/walterdalmut/.lock 3891573 NX PX 30000 OK SET /etc/lock/walterdalmut/.lock 2857152 NX PX 30000 (nil)

How do i release the lock?DEL /etc/lock/walterdalmut/.lock

How do i extend the lock?EXPIRE /etc/lock/walterdalmut/.lock 30

How do i watch for lock release?

Lock releasesSUBSCRIBE __keyspace@0__:/etc/lock/walterdalmut/.lock

1) "message" 2) "__keyspace@0__:/etc/lock/walterdalmut/.lock" 3) "del"

1) "message" 2) "__keyspace@0__:/etc/lock/walterdalmut/.lock" 3) "expired"

Thank you for listening