DevelopmentqProgrammingqData restructuringqTransactions
High Performance RedisHolistic understanding of some techniques and tricks for building out high performance Redis systems.
Agenda
TechnologyqRedis 2.xqTwemproxyqLinux
OperationsqClusteringqFailoverqOS TuningqRedis Tuning
• Software Architect with GoPro
• 10+ years building distributed systems
• 4+ years building with Redis
• Built multiple high performing Redis apps
• Previously: Apple, Netscape, Flickr
Who Am I
• Large number of clients
• High transaction rate
• Low response time
• Highly Available
System Features
Main Techniques
ü Large number of clientsü High transaction rateü Low response timeü High Availability
ü Low Response time
ü High transaction rateü Low response time
Clustering and Failover
System Tuning
Programming Techniques
• Large number of clients
• High transaction rate
• Low response time
• Pre-requisite to high availability
Features
Basic Sharding
ClientClientClientClientClient
Shard-0
Shard-1
Shard-n
• Read/Writessamenode• Uniformworkloads• Keybasedpartitioning
• Needs a uniform work load
• Great for single key operations
• Traversals are expensive
• Node loss – roughly 1/N complete outage
• Hot keys are an issue
Trade-offs
Read/Write Replicas
ClientClientClientClientClient
WriteMaster
ReadSecondary
ReadSecondary
• Read/Writessplit• HighReadworkloads• DifferentthanFailoverReplicas
• Works well for high read workload
• Replication delay
• Lose read your writes consistency
• Node loss - depends on the node
• Easy to scale up hot keys
Trade-offs
Duplicate Nodes
ClientClientClientClientWriteClient
Node-0
Node-1
ClientClientClientClientReadClient
Read
Read
1stResult
• High performance reads
• Significant client coding
• Consistency goes out the window
• Node loss – depends on your approach
• Double the cost
Trade-offs
• Clustering techniques can be combined
• Sharding + R/W Replicas
• Sharding + Multi-Node
• Multi-node is a technique of last resort
Hybrids
• Scale horizontally not vertically
• Use multiple Redis instances per host
• R/W replicas or dedicated shards for hot
keys
• Automate deployment and configuration
Scaling Up
Happily Ever After
ü LargeNumberofClientsü HighTransactionRateq HighlyAvailableq LowResponseTime
Let’sParty
Happily Ever After
ü LargeNumberofClientsü HighTransactionRateü HighlyAvailableq LowResponseTime
Let’sParty
• net.ipv4.ip_local_port_range• net.ipv4.tcp_tw_recycle• net.ipv4.tcp_tw_reuse• net.core.somaxconn
Network Tuning - Linux
• hash-max-zipmap-entries
• hash-max-zipmap-value
• list-max-ziplist-entries
• list-max-ziplist-value
• zset-max-ziplist-entries
• zset-max-ziplist-value
• set-max-intset-entries
Small Aggregate Data Switches
Happily Ever After
ü LargeNumberofClientsü HighTransactionRateü HighlyAvailableq LowResponseTime
Let’sParty
Persistent Connections• Setup is expensive
• Reduced latency
• Server tuning maybe required
Connection Management
Operation Pipelining• Substantial speedup (ops/s)
• Failure harder to manage
• Client support required
Connection Pooling• Form of persistent
connections
• Reduced latency
• Client support required
The structure of your Redis transactions can greatly effect the performance of your app.
Transaction Structure
Smaller• Lower latency• Improved responsiveness• Reduced throughput• Negative impact on AOF
Larger• Higher latency• Reduced responsiveness• Increased throughput• Less impact on AOF
Main Techniques
• Smaller Structures faster• Pops off of long lists slow• Internally “shard” data if possible
• replace strings with denser coding (bit vector, map, etc)
• Moderate sized structures
• Reduce size of modifications• Drop keys v. mutating• Reduce impact on AOF
Structured Types
Memory Usage
Isolate Modifications
Happily Ever After
ü LargeNumberofClientsü HighTransactionRateü HighlyAvailableü LowResponseTime
Let’sParty
• Allenvironmentsaredifferent• Unexpectedlatencywillmakeyoucry• Forperformance,scalehorizontally• MonitoringandToolsareEssential• Connectionsareexpensive• RedundancyandFailoverneedstobe
automatic• AOFisafairweatherfriend
Take Aways