REDIS IN A MULTI-TENANT ENVIRONMENT Iccha Sethi, Software Developer Twitter: @IcchaSethi
Jan 16, 2017
REDIS IN A MULTI-TENANT ENVIRONMENT Iccha Sethi, Software Developer
Twitter: @IcchaSethi
2
IcchaSethi.__dict__
{ “Name”: “Iccha Sethi”, “Occupation”: “Software Developer”, “Team”: “Redis”, “Company”: “Object Rocket, Rackspace”, “Fun Fact”: “loading….” }
Why does High Availability matter for Redis?
3
Why does High Availability matter for Redis?
4
Outline
• Architecture • Security and Isolation • Know thy environment
5
Outline
• Architecture • Security and Isolation • Know thy environment
6
7
Architecture
Outline
8
Redis Master
Access HA Redis: Dns: read/write port 6379 password
Outline
9
Redis Master
Redis Slave
Access HA Redis: DNS: read/write port 6379 password
Outline
10
VIP
LB1 (Master) HAProxy
Redis Master
Redis Slave
Access HA Redis: DNS: read/write port 6379 password
Outline
11
VIP
LB1 (Master) HAProxy
LB2 (Backup) HAProxy
keepalived
Redis Master
Redis Slave
Access HA Redis: DNS: read/write port 6379 password
12
Sentinels
13
Highly Available REDIS
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Slave
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
HA Create
14
Architecture - Opportunities
Opportunities
• Failover is easy • Easy to introduce changes • Reusable architecture
15
Opportunities
• Failover is easy • Easy to introduce changes • Reusable architecture
16
17
Fail Over – Redis Nodes
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Slave
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
18
Fail Over – Redis Nodes
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Slave
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
19
Fail Over – Redis Nodes
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Master
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
20
Fail Over – Redis Nodes
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Master
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
21
Fail Over – Redis Nodes
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Master
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
22
Fail Over – Redis Nodes
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Master
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
23
Fail Over – Redis Nodes
STASH NOVA
Proxy API
Sentinels
Host 0
Host 1
Host 2
Host 3
Redis Master
Redis Master
KeepAliveD HA Proxy
KeepAliveD HA Proxy
Proxy Agent
Proxy Agent
Agent
Agent
Customer :6379
Opportunities
• Failover is easy • Easy to introduce changes • Reusable architecture
24
25
Redis Resize – Vertical Scaling
STASH
Sentinels
Redis Master
Redis Slave
HA Proxy
HA Proxy
Customer :6379
26
Redis Resize – Vertical Scaling
STASH
Sentinels
Redis Master
Redis Slave
HA Proxy
HA Proxy
Customer :6379
New Redis Master
New Redis Slave
27
Redis Resize – Vertical Scaling
STASH
Sentinels
Redis Master
Redis Slave
HA Proxy
HA Proxy
Customer :6379
New Redis Master
New Redis Slave
28
Redis Resize – Vertical Scaling
STASH
Sentinels
HA Proxy
HA Proxy
Customer :6379
New Redis Master
New Redis Slave
Opportunities
• Failover is easy • Easy to introduce changes • Reusable architecture
29
Highly Available MySQL
30
31
Architecture - Challenges
Challenges
• Global sentinels • Redis drivers
32
Global Sentinels
33
34
Sentinels
Redis Drivers/Clients
35
36
Architecture - Case Study
Tale of ‘Resize Why you no work?’
37
38
Redis Resize – Vertical Scaling
STASH
Sentinels
Redis Master
Redis Slave
HA Proxy
HA Proxy
Customer :6379
New Redis Master
New Redis Slave
Outline
• Architecture • Security and Isolation • Know your environment
39
40
Security and Isolation
ACLs
41
42
VIP
LB1 (Master) HAProxy
ACLS
LB2 (Backup) HAProxy
ACLs
keepalived
Redis Master
Redis Slave
Access HA Redis: VIP: read/write port 6379
ACLs
SSL
43
44
VIP
LB1 (Master) HAProxy
ACLs SSL certs
LB2 (Backup) HAProxy
ACLs SSL certs
keepalived
Redis Master
Redis Slave
Access HA Redis: VIP: read/write port 6379 SSL port 6380
SSL
Containers
45
46
VIP
LB1 (Master) HAProxy
LB2 (Backup) HAProxy
keepalived
Redis Master
Redis Slave
Access HA Redis: VIP: read/write port 6379
Containers
47
Security and Isolation – Case Study
48
VIP
LB1 (Master) HAProxy
LB2 (Backup) HAProxy
keepalived
Redis Master
Redis Slave
Access HA Redis: VIP: read/write port 6379
Tale of the Customer with 120K connections
Outline
• Architecture • Security and Isolation • Know thy environment
49
50
Monitoring, Monitoring, Monitoring
• What do you monitor? • How often is it monitored? • Who does the monitoring? • What action is taken?
51
Monitoring, Monitoring, Monitoring
• What do you monitor? – Front door check – Connection limit – Memory usage – Failover – Failed failover – Redis process down – No valid slaves
• How often is it monitored? • Who does the monitoring? • What action is taken?
52
Monitoring, Monitoring, Monitoring
• What do you monitor? • How often is it monitored?
– Front door check – Memory usage – Failover – Failed failover – Redis process down – No valid slaves
• Who does the monitoring? • What action is taken?
53
Monitoring, Monitoring, Monitoring
• What do you monitor? • How often is it monitored? • Who does the monitoring? • What action is taken?
54
55
56
Monitoring, Monitoring, Monitoring
• What do you monitoring? • How often is it monitored? • Who does the monitoring? • What action is taken?
57
58
Monitoring, Monitoring, Monitoring
• What do you monitoring? • How often is it monitored? • Who does the monitoring? • What action is taken?
– Front door check – Memory usage – Failover – Failed failover – Redis process down – No valid slaves
59
60
Know Thy Environment - Challenges
61
62
Know Thy Environment – Case Studies
Tale of the customer saved by Zendesk
63
Outline
• Architecture • Security and Isolation • Know thy environment
64
65
66