Open source, advanced key-value store, data structure server REmote DIctionary Server. [email protected] .

open source, advanced key-value store, data structure server

http://redis.io/

REmote DIctionary Server.

[email protected]://www.yumianfeilong.com

Hello RedisInstall

$ wget http://redis.googlecode.com/files/redis-2.4.8.tar.gz $ tar xzf redis-2.4.8.tar.gz $ cd redis-2.4.8 $ make

Start Redis server & make connection[binzhang@phxrueidb04 redis-2.4.8]$ src/redis-server redis.conf[16849] 04 Mar 02:03:59 * Server started, Redis version 2.4.8[16849] 04 Mar 02:03:59 * The server is now ready to accept connections on port 6379

Connect server$ src/redis-cli redis > set player:666:name binzhang OK redis > get player:666:name "binzhang"

2

Learn More : data structure server

3

Agenda

Redis Manifesto 宣言Data structures : strings, hashes, lists, sets and

sorted sets.Leveraging RedisRedis Admin and maintenanceThe architecture of REDISCases

4

Redis Manifesto http://antirez.com/post/redis-manifesto.html

1. Redis is a DSL (Domain Specific Language) that manipulates abstract data types and implemented as a TCP daemon. keys are binary-safe strings and values are different kinds of abstract data types.

2. Redis has persistence option but Memory storage is #1. 3. The Redis API is a direct consequence of fundamental data

structures.4. Code is like a poem. 5. We believe designing systems is a fight against complexity. Most

of the time the best way to fight complexity is by not creating it at all.

6. Redis API has two levels: 1) a subset of the API fits naturally into a distributed version of Redis and 2) a more complex API that supports multi-key operations.

7. We optimize for joy. When there is no longer joy in writing code, the best thing to do is stop.

5

Redis Manifesto : where is Redis?

6

1. Redis is extremely fast, making it perfectly suited for applications that are write-heavy, data that changes often, and data that naturally fits one of Redis’s data structures (for instance, analytics data).

2. A scenario where you probably shouldn’t use Redis is if you have a very large dataset of which only a small part is “hot” (accessed often) or a case where your dataset doesn’t fit in memory.

Who is using Redis

7

Taobao Sina weibo

AgendaRedis Manifesto 宣言

Simple/fast/ Data structure : strings, hashes, lists, sets and sorted sets.

Leveraging RedisThe architecture of REDISRedis Admin and maintenanceCases

8

Data structure: Key-Value Data Store

Keys are strings which identify pieces of data (values)Values are arbitrary byte arrays that Redis doesn't care

aboutRedis is implemented as five specialized data structures

Strings, hash, list, set, sort setPub/Sub

Querying with Redis ; the above make Redis fast and easy to use, but not suitable

for every scenario

9

Data structure : KEY value

Before we dive into the specific data types, it is important to look at a few things you should keep in mind when designing the key structure that holds your data.1.a key can contain any characters, you can use separators to define a namespace with a semantic value for your business. An example might be using cache:project:319:tasks, where the colon acts as a namespace separator.2.When defining your keys, try to limit them to a reasonable size. Retrieving a key from storage requires comparison operations, so keeping keys as small as possible is a good idea. Additionally, smaller keys are more effective in terms of memory usage.3.Even though keys shouldn’t be exceptionally large, there are no big performance improvements for extremely small keys. This means you should design your keys in such a way that combines readability (to help you) and regular key sizes (to help Redis).

10

Data structure : KEY value

Redis support different kind of values1.Binary-safe strings.2.Lists of binary-safe strings.3.Hash map of strings4.Sets of binary-safe strings, that are collection of unique

unsorted elements. 5.Sorted sets, similar to Sets but where every element is

associated to a floating number score. The elements are taken sorted by score.

6.pubsub channels are a new addition to Redis 11

Data structure : strings, hashes, lists, sets , sorted sets.

1. Strings: the simplest and most basic data type (max 512M in length) 2. Redis Strings are binary safe, this means that a Redis string can contain any kind of data,

for instance a JPEG image or a serialized Ruby object.

– redis > users:leto "{name: leto, planet: dune, likes: [spice]}"

– OK

– redis > x users:leto "{name: leto, planet: dune, likes: [spice]}"

– (integer) 0redis > strlen users:leto(integer) 42redis > getrange users:leto 27 40"likes: [spice]"redis > append users:leto " OVER 9000!!"(integer) 54redis > get users:leto"{name: leto, planet: dune, likes: [spice]} OVER 9000!!"

12

The C structure sdshdr declared in sds.h represents a Redis string:

struct sdshdr { long len; long free; char buf[]; };

Data structure : strings, hashes, lists, sets , sorted sets.

• if we store a counter key, we can use commands such as INCR (or INCRBY) and DECR (or DECRBY) to increment or decrement its contained value.

– To store page visit data, we could have a key “visits:pageid:totals”redis > SET visits:2:totals 1367894OKredis > get visits:2:totals"1367894"redis > INCR visits:635:totals(integer) 1redis > INCR visits:2:totals(integer) 1367895redis > get visits:2:totals"1367895“redis > strlen visits:2:totals(integer) 7redis > incr users:leto(error) ERR value is not an integer or out of range

13

Data structure : strings, hashes, lists, sets , sorted sets.

• Much like traditional hashtables, hashes in Redis store several fields and their values inside a specific key. so they are the perfect data type to represent objects (eg: A User with a number of fields like name, surname, age, and so forth):

• Example: Designing a key namespace to store our users.redis> hset users:jdoe name "John Doe"(integer) 1redis> hmset users:jdoe email [email protected] phone "+1555313940"OKredis> hincrby users:jdoe visits 1(integer) 1redis > hget users:jdoe phone"+1555313940"

• A hash with a few fields (where few means up to one hundred or so) is stored in a way that takes very little space, so you can store millions of objects in a small Redis instance.

• Every hash can store up to 232 - 1 field-value pairs (more than 4 billion).

14

Data structure : strings, hashes, lists, sets , sorted sets.

• HDEL key field [field ...] Delete one or more hash fields• HGETALL key Get all the fields and values in a hash• HINCRBY key field increment Increment the integer value of a hash field by the given number• HKEYS key Get all the fields in a hash• HLEN key Get the number of fields in a hash• HMGET key field [field ...] Get the values of all the given hash fields• HMSET key field value [field value ...] Set multiple hash fields to multiple values• HSETNX key field value Set the value of a hash field, only if the field does not exist• HVALS key Get all the values in a hash

redis > hgetall users:jdoe1) "name"2) "John Doe"3) "email"4) "[email protected]"5) "phone"6) "+1555313940"7) "visits"8) "1"

15

Data structure : strings, hashes, lists, sets , sorted sets.

• Redis Lists are simply lists of strings, sorted by insertion order. It is possible to add elements to a Redis List pushing new elements on the head (on the left) or on the tail (on the right) of the list.

LPUSH mylist a # now the list is "a" LPUSH mylist b # now the list is "b","a" RPUSH mylist c # now the list is "b","a","c" (RPUSH was used this time)

• support for constant time insertion and deletion of elements near the head and tail, even with many millions of inserted items. Accessing elements is very fast near the extremes of the list but is slow if you try accessing the middle of a very big list, as it is an O(N) operation.

• You might want to use lists in order to implement structures such as queues • The max length of a list is 232 - 1 elements (4294967295, more than 4 billion of elements per

list).

16

Data structure : strings, hashes, lists, sets , sorted sets.

• Model a timeline in a social network, using LPUSH in order to add new elements in the user time line, and using LRANGE in order to retrieve a few of recently inserted items.

– Lrange to do paging

• You can use LPUSH together with LTRIM (O(N))to create a list that never exceeds a given number of elements, but just remembers the latest N elements.

– Capped Collections in MongoDB.

• Lists can be used as a message passing primitive,– BLPOP key [key ...] timeout Remove and get the first element in a list, or block until one is available– BRPOP key [key ...] timeout Remove and get the last element in a list, or block until one is available– LINDEX key index Get an element from a list by its index– LLEN key Get the length of a list– LPOP key Remove and get the first element in a list– RPOP key Remove and get the last element in a list– LRANGE key start stop Get a range of elements from a list– LTRIM key start stop Trim a list to the specified range

17

Data structure : strings, hashes, lists, sets , sorted sets.

• Sets are an unordered collection of Strings. It is possible to add, remove, and test for existence of members in O(1) (constant time regardless of the number of elements contained inside the Set).

• Elements in a given set can have no duplicates. this means that adding a member does not require a check if exists then add operation.

• Sets are a natural fit for circles, because sets represent collections of data, and have native functionality to do interesting things like intersections and unions.

• The max number of members in a set is 232 - 1 (4294967295, more than 4 billion of members per set).

– You can track unique things using Redis Sets. Want to know all the unique IP addresses visiting a given blog post? Simply use SADD every time you process a page view.

– Redis Sets are good to represent relations. – You can use Sets to extract elements at random using the SPOP or SRANDMEMBER commands.

18

Data structure : strings, hashes, lists, sets , sorted sets.

We want to store several circles for each of our users, so it makes sense for our key to include a bit about the user and a bit about the actual circle. (circle:jdoe:family etc)

19

redis> sadd circle:jdoe:family users:anna

redis> sadd circle:jdoe:family users:richard

redis> sadd circle:jdoe:family users:mike

(integer) 1

redis> sadd circle:jdoe:soccer users:mike

redis> sadd circle:jdoe:soccer users:adam

redis> sadd circle:jdoe:soccer users:toby

redis> sadd circle:jdoe:soccer users:apollo

(integer) 1

redis> smembers circle:jdoe:family

1) "users:richard"

2) "users:mike"

3) "users:anna"

redis> hgetall users:mike

(...)

redis> sinter circle:jdoe:family circle:jdoe:soccer

1) "users:mike"

redis> sunion circle:jdoe:family circle:jdoe:soccer

1) "users:anna"

2) "users:mike"

3) "users:apollo"

4) "users:adam"

5) "users:richard"

6) "users:toby"

Data structure : strings, hashes, lists, sets , sorted sets.

• SADD key member [member ...] Add one or more members to a set• SCARD key Get the number of members in a set• SDIFF key [key ...] Subtract multiple sets• SDIFFSTORE destination key [key ...] Subtract multiple sets and store the resulting set in a

key• SINTER key [key ...] Intersect multiple sets• SISMEMBER key member Determine if a given value is a member of a set• SMEMBERS key Get all the members in a set• SMOVE source destination member Move a member from one set to another• SPOP key Remove and return a random member from a set• SRANDMEMBER key Get a random member from a set• SREM key member [member ...] Remove one or more members from a set • SUNION key [key ...] Add multiple sets• SUNIONSTORE destination key [key ...] Add multiple sets and store the resulting set in a key

20

Data structure : strings, hashes, lists, sets , sorted sets.• Use set to implement tags

– sadd news:1000:tags 1 – (integer) 1 – sadd news:1000:tags 2 – (integer) 1 – sadd news:1000:tags 5 – (integer) 1 – sadd news:1000:tags 77 – (integer) 1 – sadd tag:1:objects 1000 – (integer) 1– sadd tag:2:objects 1000 – (integer) 1 – sadd tag:5:objects 1000 – (integer) 1 – sadd tag:77:objects 1000– (integer) 1

21

To get all the tags for a given object :redis>smembers news:1000:tags1. 5 2. 1 3. 77 4. 2

we may want the list of all the objects having as tags 1, 2, 10, and 27 at the same time

Sinter tag:1:objects tag:2:objects tag:10:objects tag:27:objects

Set: Wildcard autocomplete

• Split every username to three letter chunks– Simonw => sim, imo, mon, onw

• Create a set for each chunk– Sim => { simonw, asimov, fasim}

• If the user types “simo”, return the intersection of the “sim” and “imo”.

22

Data structure : strings, hashes, lists, sets , sorted sets.

1. Every member of a Sorted Set is associated with score, that is to sort set, from the smallest to the greatest score. members are unique, scores may be repeated.

2. With sorted sets you can add, remove, or update elements in a very fast way (in a time proportional to the logarithm of the number of elements, O(log(N))).

3. Get ranges by score or by rank (position) in a very fast way. – ZADD can be used both to add items to the set and to update the score of an existing member. The ZRANGE family of commands return items by their index position within the ordered set. The optional

WITHSCORES argument returns the score for each item in the same response. ZRANGEBYSCORE query the ordered set by score, instead of by index.

zadd friends:leto 100 ghanima 95 paul 95 chani 75 jessica 1 vladimir

redis > zrange friends:leto 0 -1 withscores 1) "vladimir" 2) "1" 3) "jessica" 4) "75" 5) "chani" 6) "95" 7) "paul" 8) "95" 9) "ghanima"10) "100"

23

Data structure : strings, hashes, lists, sets , sorted sets.Zset as index : any time you need to look up data based on range queries, you should be storing it in a sorted set.

They're indexes that you have to maintain yourself.redis > zadd hackers 1940 "Alan Kay" 1953 "Richard Stallman" 1969 "Linus Torvalds" 1912 "Alan Turing"

redis > zrange hackers 0 -11) "Alan Turing"2) "Alan Kay"3) "Richard Stallman"4) "Linus Torvalds“

redis > zrangebyscore hackers 1950 19901) "Richard Stallman"2) "Linus Torvalds“

redis > zrangebyscore hackers -inf 19501) "Alan Turing"2) "Alan Kay"

redis > zremrangebyscore hackers 1940 1960(integer) 2redis > zrange hackers 0 101) "Alan Turing"2) "Linus Torvalds"

24

Sort set : Prefix autocomplete

• Type “binz”, return “binzhang”1. Turn the first 4 or 5 characters of the strings into an integer (you can

imagine every char as a digit of a radix 256 number for instance, but there are better representation) and add all your usernames into a sorted set with score=integer.

2. Then using ZRANGEBYSCORE you can get all the elements between a given range. --ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]

3. This method is much more scalable as it's an O(log(N)) thing. (zrangebyscore is O(log(N)+M) )

• 25

sorted sets : Inverted-Index Text Search with Redis

1. an inverted index - a bunch of sets mapping terms to document IDs. 2. Assign each document an ID3. Apply stemming and stopwords first4. Create an inverted index, with one set per word5. Create a set of docIDs for each term

– ZINTERSTORE destination-zset number-of-zsets-to-intersect zset1 [zset2 ...] [WEIGHTS weight1 [weight2 ...]] [AGGREGATE SUM | MIN | MAX]

26

Learn More Data structure : Pub/Sub

• Redis has native support for the publish/subscribe (or pub/sub) pattern1. receivers subscribe to messages that match a specific

pattern (for instance, messages that are sent to a specific “channel”),

2. an procedurer/emitter to send messages to me3. emitter and receivers to be loosely coupled-- hey don’t

need to know each other.

• The pub/sub command1. PSUBSCRIBE pattern [pattern ...] Listen for messages published to channels matching the given patterns2. PUBLISH channel message Post a message to a channel3. PUNSUBSCRIBE [pattern [pattern ...]] Stop listening for messages posted to channels matching the given patterns4. SUBSCRIBE channel [channel ...] Listen for messages published to the given channels 5. UNSUBSCRIBE [channel [channel ...]] Stop listening for messages posted to the given channels

27

Learn More Data structure : Pub/Sub

redis > subscribe irc:footballReading messages... (press Ctrl-C to quit)1) "subscribe"2) "irc:football"3) (integer) 11) "message"2) "irc:football"3) "have a good day"1) "message"2) "irc:football"3) "water"

28

redis > PUBLISH irc:football "Rock you"(integer) 0redis > PUBLISH irc:football "have a good day"(integer) 1redis > PUBLISH irc:football "water"

AgendaRedis Manifesto 宣言Data structure : strings, hashes, lists, sets and

sorted sets.Leveraging RedisThe architecture of REDISRedis Admin and maintenanceCases

29

Leveraging Redis1. Operations on KEYS2. Big O Notation3. Sort4. EXPIRE5. Transaction6. Optimistic locking using check-and-set ( select for update)7. Pipelining ( commands in batch)

30

Keys operation KEYS pattern Lists all the keys in the current database that match the given pattern.

[slow] TYPE key-name Tells the type of the key. Possible types are: string, list, hash, set, zset, and

none. MONITOR Outputs the commands received by the Redis server in real time. [debug

purpose only]

KEYS h*llo KEYS h?llo KEYS h[ae]llo.

redis > type circle:jdoe:soccerSet

DEL/EXISTS/EXPIRE/TTL/PERSIST/RANDOMKEY/RENAME

Redis can handle up to 2^32 keys 31

Big O NotationHow fast a command is based on the number of items we are dealing with.O(1): fastest, Whether we are dealing with 5 items or 5 million, you'll get the same performance.

Sismember : if a value belongs to a set

O(N) : linear commands, fts etc Keys ltrim, N is the number of elements being removed.

O(log(N)): zadd is a O(log(N)) command, where N is the number of elements already in the

set.

O(log(N)+M) zremrangebyscore : N is the number of total elements in the set and M is the

number of elements to be removed.

O(N+M*log(M)) sort

32

sort• Sort the values within a list, set or sorted set.

redis > rpush users:leto:guesses 5 9 10 2 4 10 19 2(integer) 8redis > sort users:leto:guesses1) "2"2) "2"3) "4"4) "5"5) "9"6) "10"7) "10"8) "19"redis 9> sadd friends:ghanima leto paul chani jessica alia duncan(integer) 6redis > sort friends:ghanima limit 0 3 desc alpha1) "paul"2) "leto"3) "jessica"

• Redis is single-thread, so sort on salve if large dataset• Sort can store result to a key, one pattern for paginating through expensive sort results

(millions of items, for example) is to save the result to a temporary key, set an expiry on it and use that for pagination via the LRANGE command.

33

TransactionEvery Redis command is atomic, including the ones that do multiple things.

– incr is essentially a get followed by a set– getset sets a new value and returns the original– setnx first checks if the key exists, and only sets the value if it does not– Msetnx fails if any key already exist (less important, now we’ve hash)

MULTI command can run multiple commands as an atomic group.1. Marks the start of a transaction block. Subsequent commands will be queued for atomic execution using EXEC.

DISCARD can be used in order to abort a transaction. In this case, no commands are executed and the state of the connection is restored to normal.

2. The commands will be executed in order3. The commands will be executed as a single atomic operation (without another client's command being executed halfway

through)4. That either all or none of the commands in the transaction will be executed

redis > multiOKredis > hincrby groups:1percent balance -9000000000QUEUEDredis > hincrby groups:1percent balance -9000000000QUEUEDredis > exec1) (integer) -90000000002) (integer) -18000000000

34

Optimistic locking using check-and-set

• WATCH / UNWATCH– keys are monitored in order to detect changes against them. If at least one watched key is modified before

the EXEC command, the whole transaction aborts, and EXEC returns a Null multi-bulk reply to notify that the

transaction failed.• let's suppose Redis doesn't have INCR

WATCH mykey val = GET mykey val = val + 1 MULTI SET mykey $val EXEC

• Using WATCH to implement ZPOPWATCH zset element = ZRANGE zset 0 0 MULTI ZREM zset element EXEC

35

Expiration

1. Redis allows you to mark a key for expiration. 2. You can give it an absolute time in the form of a Unix timestamp (seconds since January 1, 1970)

or a time to live in seconds. a. expire pages:about 30 -- delete key after 30 secondsb. expireat pages:about 1356933600 -- delete key at 12:00 a.m. December 31st, 2012.

c. ttl pages:about -- check ttld. persist pages:about -- remove expire limit

e. setex pages:about 30 '<h1>about us</h1>’ --set a string and specify a expire time

3. Lazy Expiration algorithmKeys are expired simple when some clients tries to access a key and the key is found to be time out

4. Once every second,I. Tests 100 random keys from expired keys set.II. Deletes all the keys found expired.III. If more than 25 keys were expired, it starts again from

36

What if no available memory

Redis will return an error on write operations, but read-only query still works Can specify “maxmemory” to define a hard limit for memory usage maxmemory-policy: specify the algorithm to use when we need to reclaim memory

a. volatile-lru (default) remove a key among the ones with an expire set, trying to remove keys not recently used.b. volatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.c. volatile-random remove a random key among the ones with an expire set.d. allkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.e. allkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.

LRU and minimal TTL algorithmsa) are not precise algorithmsb) for default Redis will check three keys(“maxmemory-samples”) and pick the one that was used less recently

37

redis > set newkey "maxsize"(error) ERR command not allowed when used memory > 'maxmemory'

Redis Pipelining(How)Send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step.

$ (echo -en "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379 +PONG +PONG +PONG

38

Not paying the cost of RTT for every call;

Client: INCR XClient: INCR XClient: INCR XClient: INCR XServer: 1Server: 2Server: 3Server: 4

Redis Pipelining(why we need)

• Redis is a TCP server using the client-server model and what is called a Request/Response protocol.– The client sends a query to the server, and reads from the socket, usually in a

blocking way, for the server response.– The server processes the command and sends the response back to the client.

39

So for instance a four commands sequence is something like this:Client: INCR XServer: 1Client: INCR XServer: 2Client: INCR XServer: 3Client: INCR XServer: 4

Network Round Trip: Latency?

AgendaRedis Manifesto 宣言Data structure : strings, hashes, lists, sets and sorted

sets.Leveraging Redis

Keys/O(n)/sort/expire/transaction/watch

Redis Admin and maintenance Select database Monitor Redis Configure Persistence Starting a Redis Slave Handling a Dataset larger than memory Upgrade Redis BackUp Redis Sharding Redis Benchmarks

The architecture of REDISCases 40

Databases in Redis

1.A database contains a set of data. 2.A database is to group all of an application's data together

and to keep it separate from another application's.3.databases are simply identified by a number with the

default database being number 0.4.Number of databases is set via “databases” param in config5.change to a different database via select command

–redis [1]> select 1–OK–redis [1]> select 0–OK–redis >

41

Monitor Redis1. MONITOR , is actually part of the Redis replication system. If you telnet directly to Redis and type monitor, you'll see a live dump of all commands executing against the

database. This is really useful for debugging.2. config set slowlog-log-slower-than 03. Redis-stat : similar like prstat4. Info command

– redis > info– redis_version:2.4.8– redis_git_sha1:00000000– redis_git_dirty:0– arch_bits:64– multiplexing_api:epoll– gcc_version:4.1.2– process_id:5898– uptime_in_seconds:163519– uptime_in_days:1– lru_clock:1033766– used_cpu_sys:1.19– used_cpu_user:2.36– used_cpu_sys_children:0.00– used_cpu_user_children:0.00– connected_clients:2– connected_slaves:1– client_longest_output_list:0– client_biggest_input_buf:0– blocked_clients:0– used_memory:1384280– used_memory_human:1.32M– used_memory_rss:3637248– used_memory_peak:15032016– used_memory_peak_human:14.34M– mem_fragmentation_ratio:2.63– mem_allocator:jemalloc-2.2.5– loading:0– aof_enabled:0– changes_since_last_save:0– bgsave_in_progress:0– last_save_time:1331542318– bgrewriteaof_in_progress:0– total_connections_received:757– total_commands_processed:150064– expired_keys:0– evicted_keys:0– keyspace_hits:50019– keyspace_misses:3– pubsub_channels:0– pubsub_patterns:0– latest_fork_usec:1875– vm_enabled:0– role:master– slave0:10.14.141.15,31424,online– db0:keys=8,expires=0– db1:keys=1,expires=0

42

Redis Admin and maintenance: Configure Persistence(1)

Persistence Mode: snapshotting and AOF. It should be configured in a way that suits your dataset and usage patterns.

• snapshotting, which consists of saving the entire database to disk in the RDB format (a compressed database dump). This can be done periodically at set times, or every time a configurable number of keys changes.

# save <seconds> <changes>save 900 1 --after 900 sec (15 min) if at least 1 key changedsave 300 10 --after 300 sec (5 min) if at least 10 keys changedsave 60 10000 --after 60 sec if at least 10000 keys changed

1. The alternative is using an Append Only File (AOF). This might be a better option if you have a large dataset or your data doesn’t change very frequently../redis-server --appendonly yes

2. It is possible to combine both AOF and RDB in the same instance. 3. Master ->> Slave can be a option.4. Both are sequential IO5. When Redis starts, it will read RDB or AOF to load all data into memory.

43

Redis Admin and maintenance: Configure Persistence(2)

1. Snapshotting – performs point-in-time snapshots of dataset at specified intervals. ( – a full dump of your database to disk, overwriting the previous dump only if successful.– Can manually trigger snapshotting with the SAVE and BGSAVE commands.

• BGSAVE forks the main Redis process and saves the DB to disk in the background.• SAVE performs the same operation as BGSAVE but does so in the foreground, thereby blocking your Redis server.

– are also used when performing a master -> slave synchronization. • Append Only File(AOF)

– keeps a log of the commands that change your dataset in a separate file.– an append only log. no seeks, nor corruption problems (redis-check-aof )– Appendfsync: how often the AOF gets synched to disk (fsync syscall) :

• Always (be able to group commit), every sec, and no.

1. BGREWRITEAOF rewrites the AOF to match the current database; can reduce size of AOF greatly. (For example, if you are incrementing a counter 100 times, you'll end up with a single key in your dataset containing the final value, but 100 entries in your AOF. 99 of those entries are not needed to rebuild the current state.)

44

Redis Admin and maintenance: Master --Slave

Use slave : Load balance read queires, standby, Backup ,DW queries master-slave replication natively:

A master can have multiple slaves. Slaves are able to accept other slaves connections. Redis replication is non-blocking on the master side, this means that the master will continue to serve

queries when one or more slaves perform the first synchronization. configure replication on the configuration file before starting a server

slaveof master-ip-or-hostname masterport masterauth master-password

by connecting to a running server and using the SLAVEOF command. SLAVEOF master-ip-or-hostname [masterport] CONFIG SET masterauth password

45

Handling a Dataset Larger Than Memory

• Memory (VM) since version 2.0 (deprecated after Redis 2.4) . vm-enabled yesvm-swap-file

• Allow a dataset bigger than your available RAM by swapping rarely used values to disk and keeping all the keys and the frequently used values in memory.1. The keys are always kept in memory. Values can be swapped.2. Redis server might end up blocking clients in order to fetch the values from disk.3. Slow snapshot, Redis needs to read all the values swapped to disk in order to write them to the RDB file. AOF

is better at this case.4. VM also affects the speed of replication, because Redis masters need to perform a BGSAVE when a new slave

connects.5. SSDs such as Flash is encouraged

46

Upgrading Redis

Redis can’t do online binary upgradessolution

1. starting a new Redis server in slave mode, 2. switching over the clients to the slave 3. promoting the new server to the master role.

make sure to test before doing it on your production servers.

47

Backing up Redis

• Depending on which Redis persistence model you’re using.1.With the default persistence model (snapshotting), you’re

best off using a snapshot as a backup. /* cold backup */redis-cli BGSAVECopy

2. If you’re using only AOF, you’ll have to back up your log in order to be able to replay it on startup.

BGREWRITEAOF regularlyredis-check-aof --fix filename

• backups on a slave Redis instance or Slave as a backup

48

Sharding Redis• Where is Redis Cluster

– Under development. Probably reasonable beta for summer 2012 and ship the first stable one before end of 2012.

• Have to implemented in the client library or application

– you should probably use consistent hashing.– you will not be able to perform some operations that

affect multiple keys, because those keys might be in different shards (servers).

49

Benchmarks--- How fast is Redis?• redis-benchmark utility that simulates SETs/GETs done by N clients at the same time sending M total

queries[hadoop@phxrueidb03 src]$ ./redis-benchmark -q -n 100000PING (inline): 97370.98 requests per secondPING: 101214.58 requests per secondMSET (10 keys): 66357.00 requests per secondSET: 105263.16 requests per secondGET: 103199.18 requests per secondINCR: 104493.20 requests per secondLPUSH: 104931.80 requests per secondLPOP: 104384.13 requests per secondSADD: 104931.80 requests per secondSPOP: 103950.10 requests per secondLPUSH (again, in order to bench LRANGE): 104931.80 requests per secondLRANGE (first 100 elements): 44964.03 requests per secondLRANGE (first 300 elements): 22825.84 requests per secondLRANGE (first 450 elements): 16564.52 requests per secondLRANGE (first 600 elements): 12701.64 requests per second

50

Benchmarks- Redis VS memcached

51

Benchmarks--- How fast is Redis?• Redis is a server: all commands involve network or IPC roundtrips. Cost of most operations is

precisely dominated by network/protocol management.– low latency network

• Redis commands return an acknowledgment for all usual commands. • Redis is an in-memory data store with some optional persistency options. Some persistency

option would bring latency.– huge page & SSD

• Redis is a single-threaded server. It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed.

– Redis favors fast CPUs with large caches and not many cores.

52

AgendaRedis Manifesto 宣言Data structure : strings, hashes, lists, sets and sorted

sets.Leveraging RedisRedis Admin and maintenance

Configure Persistence/Redis Slave/Handling a Dataset larger than memory

Upgrade Redis/BackUp Redis/Sharding Redis/ benchmarksThe architecture of REDIS

How Redis works Latency in Redis Memory efficiency in 2.2 Redis Security

Cases 53

How Redis worksHow a command received by a client is processed internally by Redis:1.Redis uses a single thread that manages synchronously all network connection. A thin event library has been implemented to abstract several unix system calls (epoll, select, kqueue).2.Requests are managed with commands. Using a command table and according what event is read from sockets a command handler is invoked to perform desired action.

54

Latency in Redis

• Latency induced by network and communication– use aggregated commands (MSET/MGET) and Pipelining

• Single threaded nature of Redis– a mostly single threaded design ( I/O threads in background since 2.4)– all the requests are served sequentially

• Latency generated by slow commands– a request is slow to serve all the other clients will wait for this request to be served– commands operating on many elements, like SORT, LREM, SUNION and others. For instance taking

the intersection of two big sets can take a considerable amount of time.– run all your slow queries on replciations

• Latency generated by fork– The fork operation (running in the main thread) can induce latency by itself.

• Latency induced by swapping (operating system paging)• Latency due to AOF and disk I/O

55

Memory efficient for list

adlist.h: A generic doubly linked list implementationtypedef struct list { listNode *head; listNode *tail; void *(*dup)(void *ptr); void (*free)(void *ptr); int (*match)(void *ptr, void *key); unsigned int len;} list;typedef struct listNode { struct listNode *prev; struct listNode *next; void *value;} listNode;

O(1) is cool but *prev/*next would take amounts of bytes if *value is few bytes. Ziplist (list-max-ziplist-entries 512 & list-max-ziplist-value 64)

I.Save memory by using a little more CPUII.Pack list in a single block of memoryIII.Value header holds encoding / value lengthIV.O(memory size) LPUSH / LPOPV.Good fit for small payload, limited size

56

Memory efficient for hash

• Zmap (hash-max-zipmap-entries 512 & hash-max-zipmap-value 64)– keys and values are prefixed length "objects", the lookup will take O(N) where N is the number of elements in the

zipmap and *not* the number of bytes needed to represent the zipmap.

• Other data structure also have similar improve ( sort sets, intset)

57

Skip-list for sort set

1. Consists of several levels, Each level is a sorted list2. All keys appear in level 13. If key x appears in level n, then it also appears in all levels below n4. An element in level n points (via down pointer) to the element with same key in the level below5. Each level has int_min and int_max 6. Top points to the smallest element in the highest level

58

Memory structure

lazy rehashing ： The more operation you run into an hash table that is rhashing, the more rehashing "steps" are performed, so if the server is idle the rehashing is never complete and some more memory is used by the hash table.

active rehashing ： uses 1 millisecond every 100 milliseconds of CPU time in order to help rehashing the main Redis hash table (the one mapping top-level keys to values).

59

memory fragmentation Info

used_memory:21279952 memory allocated to redisused_memory_human:20.29Mused_memory_rss:23654400 memory from OS, result of ps or topused_memory_peak:21704152used_memory_peak_human:20.70Mmem_fragmentation_ratio:1.11 = used_memory_rss/used_memorymem_allocator:jemalloc-2.2.5 default in linux 2.4 and 2.6

String: dictEntry(12bytes)+sds(store key)+redisObject(12bytes)+sds(store value)

60

Set hello word = 16(dictEtnry) + 16 (redisObject) + 16(“hello”) + 16(“world”),

Redis Security1. Redis is designed to be accessed by trusted clients inside trusted environments.

Firewall on redis port

2. Redis is not optimized for maximum security but for maximum performance and simplicity.3. Authentication feature

The password is in clear text inside redis.conf file and client configuration AUTH command, like every other Redis command, is sent unencrypted

4. Data encryption support - None5. Disabling of specific commands -- rename-command FLUSHALL ""

61

AgendaRedis Manifesto 宣言Data structure : strings, hashes, lists, sets and

sorted sets.Leveraging RedisRedis Admin and maintenanceThe architecture of REDIS

Event library /Memory efficientLatency /Security

Cases http://lloogg.com a simple Twitter clone Sina Weibo

62

CASE 1: http://lloogg.com/

1. List, lpush,ltrim show access history2. Strings:incr show pageveiws3. zset: show opt references, ref as a score

63

a simple Twitter clone

Register:INCR global:nextUserId => 1000 SET uid:1000:username antirez SET uid:1000:password p1pp0

SET username:antirez:uid 1000

64

Circles and posts: uid:1000:followers => Set of uids of all the followers users uid:1000:following => Set of uids of all the following user

uid:1000:posts => a List of post ids, every new post is LPUSHed here.

Cookie:

SET uid:1000:auth fea5e81ac8ca77622bed1c2132a021f9 SET auth:fea5e81ac8ca77622bed1c2132a021f9 1000

New post

INCR global:nextPostId => 10343 SET post:10343 "$owner_id|$time|I'm having fun with Retwis"

LPUSH to user’s followers

foreach($followers as $fid) { $r->push("uid:$fid:posts",$postid,false); }

Push to latest news:

$r->push("global:timeline",$postid,false); $r->ltrim("global:timeline",0,1000);

Pageing

$posts = $r->lrange($key,$start,$start+$count);

Making it horizontally scalable1.Split by hash key

Sina weibo

65

Summary

Redis Manifesto Memory #1;data structure server

Data structure : strings, hashes, lists, sets and sorted sets, pub sub

Leveraging Redis KEYS/Big O /Sort/EXPIRE/Transaction/Optimistic locking/Pipelining

Redis Admin and maintenance Select /Redis/Persistence/Replication/ VM/Upgrade/BackUp/Sharding /Benchmarks

The architecture of REDIS Event library /Memory efficient Latency /Security

Cases http://lloogg.com / a simple Twitter clone /Sina Weibo

66

Redis

• Q&A

67