Planning for the Horizontal: Scaling Node.js Applications

TheReddest [email protected]

PLANNING FOR THE HORIZONTAL

SCALING NODE.JS APPLICATIONS

Brandon Cannaday

Thursday, April 4, 13

ME

HOSTING, DATA, STATS FOR NODE.JS

modulus.io

Thursday, April 4, 13

WHEN TO SCALE?

1. RESPONSE TIMES

2. CPU USAGE

3. CONCURRENT CONNECTIONS

Thursday, April 4, 13

NODEFLY.COM

Thursday, April 4, 13

STARTING POINT

mydomain.com

SERVER

> NODE

Thursday, April 4, 13

NODE TWEAKS

http.globalAgent.maxSockets = Number.MAX_VALUE;

CONCURRENT OUTGOING CONNECTION LIMIT

Thursday, April 4, 13

HURTLE: THE SERVER

LINUX CONFIGURATION

1. FILE-MAX

2. SOMAXCONN

3. ULIMIT

Thursday, April 4, 13

FILE-MAX

SYSTEM FILE DESCRIPTOR LIMIT

1. Run sysctl -w fs.file-max=65535

2. Run sysctl -p

Thursday, April 4, 13

SOMAXCONN

SOCKET LISTEN QUEUE LENGTH

1. Run sysctl -w net.core.somaxconn=65535

2. Run sysctl -p

Thursday, April 4, 13

ULIMIT

PER PROCESS FILE DESCRIPTOR LIMIT

1. Edit /etc/security/limits.conf

2. Add the following:

* soft nofile 65535* hard nofile 65535root soft nofile 65535root hard nofile 65535

Thursday, April 4, 13

RUNNING SMOOTH

mydomain.com

SERVER

> NODE

Thursday, April 4, 13

HURTLE: THE CPU

BUY A BIGGER BOX

SERVER

> NODE

SERVER

> NODE

1 CORE

4 CORES

Thursday, April 4, 13

MULTICORE NODE

1 2 3 4

100%

CORE

USAGE

Thursday, April 4, 13

CLUSTER MODULE

SERVER

> NODE > NODE

> NODE > NODE

mydomain.com

Thursday, April 4, 13

CLUSTER EXAMPLE

var cluster = require('cluster');var http = require('http');var numCPUs = require('os').cpus().length;

if(cluster.isMaster) { for(var i = 0; i < numCPUs; i++) { cluster.fork(); }}else { http.createServer(function(req, res) { res.writeHead(200); res.end('Hello World.'); }).listen(80);}

The Cluster Module

Thursday, April 4, 13

CLUSTER EXAMPLE

var cluster = require('cluster');var http = require('http');var numCPUs = require('os').cpus().length;

if(cluster.isMaster) { for(var i = 0; i < numCPUs; i++) { cluster.fork(); }}else { http.createServer(function(req, res) { res.writeHead(200); res.end('Hello World.'); }).listen(80);}

Fork Children

Thursday, April 4, 13

CLUSTER EXAMPLE

var cluster = require('cluster');var http = require('http');var numCPUs = require('os').cpus().length;

if(cluster.isMaster) { for(var i = 0; i < numCPUs; i++) { cluster.fork(); }}else { http.createServer(function(req, res) { res.writeHead(200); res.end('Hello World.'); }).listen(80);}

Handle Requests

Thursday, April 4, 13

CLUSTER LISTEN

WORKER

listen(...)

MASTER

Handle

Thursday, April 4, 13

ROLLING UPDATES

1. UPDATE SCRIPT

2. WORKER -> STOP LISTENING

3. KILL WORKER

4. CALL FORK() AGAIN

Thursday, April 4, 13

CLUSTER MODULE

SERVER

> NODE > NODE

> NODE > NODE

mydomain.com

Thursday, April 4, 13

HURTLE: SHARED STATE

SERVER

> NODE > NODE

> NODE > NODE

NO SHARED STATE

Thursday, April 4, 13

INSTALL REDIS

SERVER

> NODE > NODE

> NODE > NODE

REDIS

Thursday, April 4, 13

EXAMPLE 1: SESSION

var express = require('express'), app = express();

app.use(express.cookieParser());app.use(express.session({ secret: 'My Cookie Signing Secret'}));

app.get('/', function(req, res) { req.session.somekey = 'some value';});

MEMORY STORE

Thursday, April 4, 13

EXAMPLE 1: SESSION

var express = require('express'), RedisStore = require('connect-redis')(express), app = express();

app.use(express.cookieParser());app.use(express.session({ store: new RedisStore({ host: 'localhost', port: 6379 }), secret: 'My Cookie Signing Secret'}));

app.get('/', function(req, res) { req.session.somekey = 'some value';});

REDIS STORE

Thursday, April 4, 13

EXAMPLE 2: SOCKET.IO

var RedisStore = require('socket.io/lib/stores/redis') , redis = require('socket.io/node_modules/redis') , pub = redis.createClient() , sub = redis.createClient() , client = redis.createClient();

io.set('store', new RedisStore({ redisPub : pub, redisSub : sub, redisClient : client}));

https://github.com/LearnBoost/Socket.IO/wiki/Configuring-Socket.IO

Thursday, April 4, 13

RUNNING SMOOTH

SERVER

> NODE > NODE

> NODE > NODE

REDIS

mydomain.com

Thursday, April 4, 13

LAST HURTLE: HORIZONTAL

APP SERVER A

> NODE > NODE

> NODE > NODE

REDIS

APP SERVER B

> NODE > NODE

> NODE > NODE

REDIS

Thursday, April 4, 13

SEPARATE REDIS

APP SERVER A

> NODE > NODE

> NODE > NODE

APP SERVER B

> NODE > NODE

> NODE > NODE

REDIS

SERVER

Thursday, April 4, 13

LOAD BALANCING

APP SERVER A

> NODE > NODE

> NODE > NODE

APP SERVER B

> NODE > NODE

> NODE > NODE

REDIS

SERVER

LOAD BALANCER

SERVER

mydomain.com

Thursday, April 4, 13

LOAD BALANCING

1. MANAGED

2. INSTALL ONE

3. WRITE YOUR OWN

Thursday, April 4, 13

NGINX

http { upstream mydomain_com { server host1.mydomain.com:80; server host2.mydomain.com:80; } server { listen 80; server_name www.mydomain.com; location / { proxy_pass http://mydomain_com; } }}

LOAD BALANCER

SERVER

Thursday, April 4, 13

WRITE ONE

https://github.com/substack/bouncy

Thursday, April 4, 13

BOUNCY

bouncy modulevar bouncy = require('bouncy');

var hosts = [ 'host1.mydomain.com', 'host2.mydomain.com'];

var count = 0;

var server = bouncy(function(req, res, bounce) {

count++; var host = hosts[count % hosts.length];

bounce(host, 80);

});

server.listen(80);

Thursday, April 4, 13

BOUNCY

Server collectionvar bouncy = require('bouncy');

var hosts = [ 'host1.mydomain.com', 'host2.mydomain.com'];

var count = 0;

var server = bouncy(function(req, res, bounce) {

count++; var host = hosts[count % hosts.length];

bounce(host, 80);

});

server.listen(80);

Thursday, April 4, 13

BOUNCY

Create server

var bouncy = require('bouncy');

var hosts = [ 'host1.mydomain.com', 'host2.mydomain.com'];

var count = 0;

var server = bouncy(function(req, res, bounce) {

count++; var host = hosts[count % hosts.length];

bounce(host, 80);

});

server.listen(80);

Thursday, April 4, 13

BOUNCY

Bounce request

var bouncy = require('bouncy');

var hosts = [ 'host1.mydomain.com', 'host2.mydomain.com'];

var count = 0;

var server = bouncy(function(req, res, bounce) {

count++; var host = hosts[count % hosts.length];

bounce(host, 80);

});

server.listen(80);

Thursday, April 4, 13

AFFINITY

SESSION AFFINITYSTICKY SESSIONS

SEND THE SAME PERSON BACK TO THE SAME SERVER

Thursday, April 4, 13

NGINX AFFINITY

http { upstream mydomain_com { sticky; server host1.mydomain.com:80; server host2.mydomain.com:80; } server { listen 80; server_name www.mydomain.com; location / { proxy_pass http://mydomain_com; } }}

Thursday, April 4, 13

CUSTOM AFFINITY

req.headers['x-forwarded-for']

req.connection.remoteAddress

Thursday, April 4, 13

RUNNING SMOOTH

APP SERVER A

> NODE > NODE

> NODE > NODE

APP SERVER B

> NODE > NODE

> NODE > NODE

REDIS

SERVER

LOAD BALANCER

SERVER

mydomain.com

Thursday, April 4, 13

ROLLING UPDATES

1. REMOVE APP SERVER FROM LOAD BALANCER

2. UPGRADE APP SERVER

3. ADD BACK

4. REPEAT

Thursday, April 4, 13

SSL

TERMINATE EARLY

Thursday, April 4, 13

SSL

APP SERVER A

> NODE > NODE

> NODE > NODE

APP SERVER B

> NODE > NODE

> NODE > NODE

REDIS

SERVER

LB

SERVER

SSL SSL TERMINATOR

Thursday, April 4, 13

SSL

LB

SERVER

SSL

mydomain.com

80 443

Thursday, April 4, 13

STUD

https://github.com/bumptech/stud

frontend = [*]:443

backend = [127.0.0.1]:80

ssl = on

pem-file = "myCert.pem"

EXAMPLE CONFIG FILE

Thursday, April 4, 13

RUNNING SMOOTH W/SSL

APP SERVER A

> NODE > NODE

> NODE > NODE

APP SERVER B

> NODE > NODE

> NODE > NODE

REDIS

SERVER

LB

SERVER

SSLmydomain.com

Thursday, April 4, 13

HUGE

REDIS

SERVER

LB

SERVER

SSL LB

SERVER

SSL

Thursday, April 4, 13

DNS

ROUND-ROBIN DNSMULTIPLE RECORDS,

ONE DOMAIN

Thursday, April 4, 13

ROUND-ROBIN DNS

CLIENT 1 1. xxx.xxx.xxx.x2. xxx.xxx.xxx.y

CLIENT 2 1. xxx.xxx.xxx.y2. xxx.xxx.xxx.x

Thursday, April 4, 13

RUNNING SMOOTH

REDIS

SERVER

LB

SERVER

SSL LB

SERVER

SSL

Thursday, April 4, 13

BIG ENOUGH

SERVER

> NODE

Thursday, April 4, 13

BIG ENOUGH

SERVER

> NODE

Thursday, April 4, 13