HIgh Performance Redis- Tague Griffith, GoPro

High Performance RedisTague Griffith

DevelopmentqProgrammingqData restructuringqTransactions

High Performance RedisHolistic understanding of some techniques and tricks for building out high performance Redis systems.

Agenda

TechnologyqRedis 2.xqTwemproxyqLinux

OperationsqClusteringqFailoverqOS TuningqRedis Tuning

Introductions

• Software Architect with GoPro

• 10+ years building distributed systems

• 4+ years building with Redis

• Built multiple high performing Redis apps

• Previously: Apple, Netscape, Flickr

Who Am I

Who Are You?

Loki The Wolfdog

What is High Performance

• Large number of clients

• High transaction rate

• Low response time

• Highly Available

System Features

Main Techniques

ü Large number of clientsü High transaction rateü Low response timeü High Availability

ü Low Response time

ü High transaction rateü Low response time

Clustering and Failover

System Tuning

Programming Techniques

Clustering

• Large number of clients

• High transaction rate

• Low response time

• Pre-requisite to high availability

Features

Basic Sharding

ClientClientClientClientClient

Shard-0

Shard-1

Shard-n

• Read/Writessamenode• Uniformworkloads• Keybasedpartitioning

• Needs a uniform work load

• Great for single key operations

• Traversals are expensive

• Node loss – roughly 1/N complete outage

• Hot keys are an issue

Trade-offs

Read/Write Replicas

ClientClientClientClientClient

WriteMaster

ReadSecondary

ReadSecondary

• Read/Writessplit• HighReadworkloads• DifferentthanFailoverReplicas

• Works well for high read workload

• Replication delay

• Lose read your writes consistency

• Node loss - depends on the node

• Easy to scale up hot keys

Trade-offs

Duplicate Nodes

ClientClientClientClientWriteClient

Node-0

Node-1

ClientClientClientClientReadClient

Read

Read

1stResult

• High performance reads

• Significant client coding

• Consistency goes out the window

• Node loss – depends on your approach

• Double the cost

Trade-offs

• Clustering techniques can be combined

• Sharding + R/W Replicas

• Sharding + Multi-Node

• Multi-node is a technique of last resort

Hybrids

• Scale horizontally not vertically

• Use multiple Redis instances per host

• R/W replicas or dedicated shards for hot

keys

• Automate deployment and configuration

Scaling Up

Happily Ever After

ü LargeNumberofClientsü HighTransactionRateq HighlyAvailableq LowResponseTime

Let’sParty

Failover

• Twemproxy

• Sentinel

Out of the Box Failover Systems

Happily Ever After

ü LargeNumberofClientsü HighTransactionRateü HighlyAvailableq LowResponseTime

Let’sParty

System Tuning

• Network tuning

• Persistence settings

• Small Aggregate Data settings

Tuning

• net.ipv4.ip_local_port_range• net.ipv4.tcp_tw_recycle• net.ipv4.tcp_tw_reuse• net.core.somaxconn

Network Tuning - Linux

• tcp-backlog

• tcp-keepalive

Network Tuning - Redis

• hash-max-zipmap-entries

• hash-max-zipmap-value

• list-max-ziplist-entries

• list-max-ziplist-value

• zset-max-ziplist-entries

• zset-max-ziplist-value

• set-max-intset-entries

Small Aggregate Data Switches

• RDB

• Fork and Save

• AOF

• On “client” thread

• Fsync policy

Persistence

Happily Ever After

ü LargeNumberofClientsü HighTransactionRateü HighlyAvailableq LowResponseTime

Let’sParty

Programming Techniques

• Connection Management

• Transaction Structure

• Data Layout

Key Areas to Consider

Persistent Connections• Setup is expensive

• Reduced latency

• Server tuning maybe required

Connection Management

Operation Pipelining• Substantial speedup (ops/s)

• Failure harder to manage

• Client support required

Connection Pooling• Form of persistent

connections

• Reduced latency

• Client support required

The structure of your Redis transactions can greatly effect the performance of your app.

Transaction Structure

Smaller• Lower latency• Improved responsiveness• Reduced throughput• Negative impact on AOF

Larger• Higher latency• Reduced responsiveness• Increased throughput• Less impact on AOF

Main Techniques

• Smaller Structures faster• Pops off of long lists slow• Internally “shard” data if possible

• replace strings with denser coding (bit vector, map, etc)

• Moderate sized structures

• Reduce size of modifications• Drop keys v. mutating• Reduce impact on AOF

Structured Types

Memory Usage

Isolate Modifications

Happily Ever After

ü LargeNumberofClientsü HighTransactionRateü HighlyAvailableü LowResponseTime

Let’sParty

Hey,WhataboutMySystem?

• Allenvironmentsaredifferent• Unexpectedlatencywillmakeyoucry• Forperformance,scalehorizontally• MonitoringandToolsareEssential• Connectionsareexpensive• RedundancyandFailoverneedstobe

automatic• AOFisafairweatherfriend

Take Aways

Questions?

Thanks!