Alternative Data Structures in Ruby

Alternative Data Structures in Ruby

Tyler McMullen

Friday, February 19, 2010

Why?


Why?

• Speed

• Memory

• Clarity


What’s wrong with my favorite data structure, X?


Nothing. (Maybe.)


•Bloom Filter

•BK-tree

•Splay Tree

•Trie


Bloom Filters

• Tests for existence in a set

• Probabilistic

• Minimal memory use


100 million strings in a Set

Traditional Set: Minimum 10gb



Traditional Set: Minimum 10gbBloom Filter (0.00001): 280mb



Traditional Set: Minimum 10gbBloom Filter (0.00001): 280mb

Bloom Filter (0.001): 170mb



10 2 3 4 5 6 7


10 2 3 4 5 6 7

“to be or not to be”


10 2 3 4 5 6 7

add: “to be or not to be”


10 2 3 4 5 6 7

add: “that is the question”


10 2 3 4 5 6 7

query: “whether ‘tis nobler”

NO MATCH


10 2 3 4 5 6 7

query: “to be or not to be”

MATCH


10 2 3 4 5 6 7

query: “in the mind to suffer”

FALSE MATCH


File Server


File Server

Request

exists?

200 404

Y N


File Server

Request

exists?

200 404

Y N

Bloom Filter


Bloom Filter

• Test for existence in set

• Tiny Memory Footprint

• Excellent Speed


BK-tree


BK-tree

• find items within a distance of a target

• reduces search space

• works inside a metric space


Triangle Inequality| d(x, y) - d(x, z) | ≤ d(y, z)



x

y

z



1

4

x

y

z



1

4

x

y

z

?


Triangle Inequality| 4 - 1 | ≤ d(y, z)

1

4

x

y

z

?


Triangle Inequality3 ≤ d(y, z)

1

4

x

y

z

≥3


BK-tree

paste

pasta

taser

pastor

shave

light


BK-tree

paste

pasta

taser

pastor

shave

light


BK-tree

paste

pasta taserpastor shave light1 2 3 4 5

root


BK-tree

paste


rootpastu


BK-tree

paste


rootpastu

1


BK-tree

paste


rootpastu

1


BK-tree

paste

pasta pastor

rootpastu

1

1 2


BK-tree

paste

pasta pastor

rootpastu

1

1 2


BK-tree

paste


root


BK-tree

paste


rootpastu


BK-tree

paste


rootpastu


BK-tree

paste


rootpastu


BK-tree

paste


rootpastu


BK-tree

• Most often used for spelling correctors

• Work in any metric space

• Reduce the search space


Splay Tree


Tangent: Access Patterns


Access Patterns

Usually assumed to be random or even.


Access Patterns

Rarely the case.


Splay Tree

• Self-balancing binary tree

• Brings most accessed items toward root

• The more uneven the access pattern, the better


Splay Tree

7

4

2 6

5 41 3

11

9 13

12 148 10


Splay Tree

7

4

2 6

5 41 3

11

9 13

12 148 10


Splay Tree

7

4

2 6

5 41 3

11

9

13

12 14

8

10


Splay Tree

7

4

2 6

5 41 3

11

9

13

12 14

8

10


Splay Tree

• Made for very uneven access patterns

• Caches, Garbage collectors, etc...


Trie


Trie

• O(1) on lookup, add, removal

• Ordered traversals

• Prefix matching

• Excellent memory usage (depending on implementation)


Trie


Trie

T

H

N

I

add: “thin”


Trie

T

H

N

I

R

A

P

add: “trap”


Trie

T

H

N

I

R

A

P

B

A

R

add: “bar”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

add: “burp”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “trap”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “trap”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “trap”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “trap”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “trap”

Success!Friday, February 19, 2010

Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “bumpkin”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “bupkis”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “bupkis”


Trie

T

H

N

I

R

A

P

B

A

R

U

R

P

query: “bupkis”

Fail!Friday, February 19, 2010

Trie

Example: Autocompleter


Trie

class Autocompleter def initialize(words) @trie = Trie.new words.each { |word| @trie.add(word) } end

def query(word) return @trie.children(word) endend


Trieclass Autocompleter def initialize(words) @trie = Trie.new words.each { |word| @trie.add(word) } end

def call(env) request = Rack::Request.new(env) return [200, { ‘content-‐type’ => ‘application/json’ }, @trie.children(word).to_json] endend


Conclusion: Data structures are cool.


Questions?


Alternative Data Structures in Ruby

Documents

r t h r

trapb u r p

bupkisb u r p

burpb u r p

bumpkinb u r p

trapt h r

p i n success

p i n fail