An Experimental Analysis of Diﬀerent Key-Value Stores and … · 2017-03-08 · Key-Value Stores and Relational Databases David Gembalczyk Felix Martin Schuhknecht Jens Dittrich

An Experimental Analysis of DifferentKey-Value Stores

andRelational Databases

David GembalczykFelix Martin Schuhknecht

Jens Dittrich

BTW 2017

Information Systems Group, Saarland Universityinfosys.uni-saarland.de

An Experimental Analysis of DifferentKey-Value Stores

andRelational Databases

David GembalczykFelix Martin Schuhknecht

Jens Dittrich

BTW 2017

Information Systems Group, Saarland Universityinfosys.uni-saarland.de

How different are they really?

2/20

OLTP OLAP

2/20

Range QueriesPoint Queries

OLTP OLAP

2/20


Single Entry All Entries

OLTP OLAP

2/20



Relational DBs

OLTP OLAP

2/20



Relational DBs Relational DBs

OLTP OLAP

2/20




OLTP OLAP

Key-Value Stores

2/20




OLTP OLAP

Key-Value Stores Key-Value Stores

2/20

Systems

3/20

Systems

3/20

Systems

3/20

Systems

3/20

Systems

3/20

HyPer(Demo 0.5)

( )

Systems

3/20

Relational Systems

4/20

Relational Systems

Primary Key A B C

1 a1 b1 c1

2 a2 b2 c2

Relational Table

4/20

Relational Systems

Primary Key A B C

1 a1 b1 c1

2 a2 b2 c2

Relational Table

PostgreSQL, MonetDB,

Hyper

4/20

Relational Systems

Primary Key A B C

1 a1 b1 c1

2 a2 b2 c2

Relational Table

PostgreSQL, MonetDB,

Hyper

N_NATIONKEY(integer) N_NAME(char(25)) N_REGIONKEY(integer) N_COMMENT(varchar(152)))

123 “Germany” 5 “some comment”

... ... ... ...

TPC-H NATION

4/20

Redis

“any string”Key Value

“any string”

5/20

Redis

“any string”Key

[ “a”, “b”, “c”, “d”]Value

5/20

Redis


{ ( 0.5, „a“ ), { ( 1.0, „c“ ), { ( 1.5, „b“ ), { ( 2.0, „d“ ) }

5/20

Redis


[ “1” ➞ “a“ , [ “2” ➞ “b“, [ “3” ➞ “c“, [ “4” ➞ “d“ ]

5/20

Redis


[ “1” ➞ “a“ , [ “2” ➞ “b“, [ “3” ➞ “c“, [ “4” ➞ “d“ ]

“NATION_123”Key

Value

[ N_NAME ➞ “Germany“ , [ N_REGIONKEY ➞ “5“, [ N_COMMENT ➞ “some comment“ ]

TPC-H NATION

5/20

Aerospike

6/20

Aerospike

“any string 1”Key

6/20

Aerospike


Value

Bin

“binA”: “a”Bin

“binB”: 42Bin

“binC”: [“a”, “b”, “c”]

6/20

Aerospike


Value

Bin


“binB”: 42Bin

“binC”: [“a”, “b”, “c”]

Value

Bin

“binB”: “b”Bin

“sub”: {“binD”: “d”, “binE”: “e”}“any string 2”Key

6/20

Aerospike


Value

Bin


“binB”: 42Bin

“binC”: [“a”, “b”, “c”]

Column?

Value

Bin



6/20

Aerospike


Value

Bin


“binB”: 42Bin

“binC”: [“a”, “b”, “c”]

Column?

Value

Bin



6/20

Aerospike


Value

Bin


“binB”: 42Bin

“binC”: [“a”, “b”, “c”]

SetColumn?

Value

Bin



6/20

Aerospike


Value

Bin


“binB”: 42Bin

“binC”: [“a”, “b”, “c”]

SetColumn?

Table?

Value

Bin



6/20

Aerospike


Value

Bin


“binB”: 42Bin

“binC”: [“a”, “b”, “c”]

SetColumn?

Table?

Value

Bin



123Key

Value

Bin

N_NAME: “Germany”

Bin

N_REGIONKEY: 5

Bin

N_COMMENT: “some comment”

TPC-H NATIONSet

6/20

PostgreSQL

7/20

PostgreSQLHSTORE

Key Value

1

[ “A” ➞ “a1”, “B” ➞ “b1”, “C” ➞ “c1”]

7/20

PostgreSQLHSTORE

Key Value

1

[ “A” ➞ “a1”, “B” ➞ “b1”, “C” ➞ “c1”]

JSONB

Key Value

1

{ “A”: “a1”, “B”: “b1”, “C”: “c1”}

2

{ “A”: “a2”, “B”: “b2”, “C”: “c2”, “sub”: { “D”: “d” “E”: “e” }}

7/20

PostgreSQLHSTORE

Key Value

1

[ “A” ➞ “a1”, “B” ➞ “b1”, “C” ➞ “c1”]

JSONB

Key Value

1

{ “A”: “a1”, “B”: “b1”, “C”: “c1”}

2

{ “A”: “a2”, “B”: “b2”, “C”: “c2”, “sub”: { “D”: “d” “E”: “e” }}

Key Value

123

[ “N_NAME” ➞ “Germany”, “N_REGIONKEY” ➞ “5”, “N_COMMENT” ➞ “some comment”]

Key Value

123

[ “N_NAME”: “Germany”, “N_REGIONKEY”: 5, “N_COMMENT”: “some comment”]

7/20

Comparing Apples and Oranges

8/20


vs

8/20


vsRAM-disk

enlarge caches

warm-up run

8/20


vsRAM-disk

enlarge caches

warm-up run

vs Custom Interface(s)

8/20


vsRAM-disk

enlarge caches

warm-up run


8/20


vsRAM-disk

enlarge caches

warm-up run


vs1 2 3 41

2

3

4

8/20

Experimental Setup

9/20

Experimental Setup

LINEITEMLINEITEMKEY

ORDERKEYPARTKEYSUPPKEY

LINENUMBERQUANTITY

EXTENDEDPRICEDISCOUNT

TAXRETURNFLAGLINESTATUSSHIPDATE

COMMITDATERECEIPTDATESHIPINSTRUCT

SHIPMODECOMMENT

PARTSUPPPARTSUPPKEY

PARTKEYSUPPKEYAVAILQTY

SUPPLYCOSTCOMMENT

modifiedTPC-H

9/20

Experimental Setup

LINEITEMLINEITEMKEY


LINENUMBERQUANTITY




SHIPMODECOMMENT

PARTSUPPPARTSUPPKEY


SUPPLYCOSTCOMMENT

modifiedTPC-H

Scale Factor 1 9/20

Experimental Setup

LINEITEMLINEITEMKEY


LINENUMBERQUANTITY




SHIPMODECOMMENT

PARTSUPPPARTSUPPKEY


SUPPLYCOSTCOMMENT

modifiedTPC-H

Scale Factor 1

warm-uprun

3 measurementruns

50,000 ops/run

9/20

Experimental Setup

LINEITEMLINEITEMKEY


LINENUMBERQUANTITY




SHIPMODECOMMENT

PARTSUPPPARTSUPPKEY


SUPPLYCOSTCOMMENT

modifiedTPC-H

Scale Factor 1

warm-uprun

3 measurementruns

50,000 ops/run

Intel Xeon X5690 Intel Xeon X5690

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

3.47 GHz

96GBRAM

96GBRAM

9/20

Round-Trip Time

jdbc.executeQuery( "SELECT 1;" ); // Relational DBsredis.echo( "1" ); // Redisaerospike.exists( non_existing_key ); // Aerospike

0 2 4 6 8 10 12 14 16 18 20

100 000

200 000

300 000

400 000

500 000

Clients

Perform

ance

[Q/sec]

PostgreSQL MonetDBHyPer (Demo) AerospikeRedis

10/20

Round-Trip Time

jdbc.executeQuery( "SELECT 1;" ); // Relational DBsredis.echo( "1" ); // Redisaerospike.exists( non_existing_key ); // Aerospike

PostgreSQL MonetDBHyPer (Demo) AerospikeRedis

10/

0 2 4 6 8 10 12 14 16 18 20

100 000

200 000

300 000

400 000

500 000

Clients

Perform

ance

[Q/sec]

PostgreSQL MonetDBAerospike RedisNO-DB YCSBCUSTOM

PostgreSQL MonetDBAerospike RedisNO-DB YCSBCUSTOM

20

Modifying the Store

Inserts

Batch Size 1

50,000 Inserts into LINEITEM and ORDERS

PG-row PG-hstore PG-jsonb MonetDB HyPer (Demo)Aerospike Redis

0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

Clients

Perform

ance

[ops/sec]

12/20

Inserts

Batch Size 1



0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

Clients

Perform

ance

[ops/sec]

12/20

Inserts

Batch Size 1



0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

Clients

Perform

ance

[ops/sec]

12/20

Inserts

Batch Size 1



0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

Clients

Perform

ance

[ops/sec]

12/20

Inserts50,000 Inserts into LINEITEM and ORDERS




Single Client

1 10 100 1 000100

1 000

10 000

100 000

Batch Size

Perform

ance

[ops/sec]

13/20

Inserts50,000 Inserts into LINEITEM and ORDERS




Single Client

1 10 100 1 000100

1 000

10 000

100 000

Batch Size

Perform

ance

[ops/sec]

13/20

Deletes50,000 Deletes in LINEITEM and ORDERS


Batch Size 1

0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

400 000

Clients

Perform

ance

[ops/sec]

14/20



Batch Size 1

0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

400 000

Clients

Perform

ance

[ops/sec]

14/20



Batch Size 1

0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

400 000

Clients

Perform

ance

[ops/sec]

14/20

Reading from the Store

Queries on Primary Key50,000 Point-Queries in LINEITEM and ORDERS

0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

400 000

Clients

Perform

ance

[ops/sec]


Batch Size 1

16/20


0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

400 000

Clients

Perform

ance

[ops/sec]


Batch Size 1

16/20


0 2 4 6 8 10 12 14 16 18 200

100 000

200 000

300 000

400 000

Clients

Perform

ance

[ops/sec]


Batch Size 1

16/20



Single Client

1 10 100 1 000

10 000

100 000

Batch Size

Perform

ance

[ops/sec]

17/20



Single Client

1 10 100 1 000

10 000

100 000

Batch Size

Perform

ance

[ops/sec]

17/20



Single Client

1 10 100 1 000

10 000

100 000

Batch Size

Perform

ance

[ops/sec]

17/20

Queries on Secondary Index

1 10 100 1 0000

20 000

40 000

60 000

80 000

100 000

Range Size

Perform

ance

[ops/sec]


SELECT MAX([ ps_supplycost | o_totalprice | l_discount ]) FROM [PARTSUPP | ORDERS | LINEITEM]

8 Clients

18/20


1 10 100 1 0000

20 000

40 000

60 000

80 000

100 000

Range Size

Perform

ance

[ops/sec]



WHERE [ ps_partkey | o_custkey | l_orderkey ] = value if Range Size = 1

8 Clients

18/20


1 10 100 1 0000

20 000

40 000

60 000

80 000

100 000

Range Size

Perform

ance

[ops/sec]




WHERE start <= [ ps_partkey | o_custkey | l_orderkey ] AND [ ps_partkey | o_custkey | l_orderkey ] <= end if Range Size ≥ 1

8 Clients

18/20


1 10 100 1 0000

20 000

40 000

60 000

80 000

100 000

Range Size

Perform

ance

[ops/sec]




WHERE start <= [ ps_partkey | o_custkey | l_orderkey ] AND [ ps_partkey | o_custkey | l_orderkey ] <= end if Range Size ≥ 1

8 Clients

18/20

Complex Analytical Queries




Q01 Q060.01

0.1

1

10

100

TPC-H Query

Duration

[sec]

19/20





Q01 Q060.01

0.1

1

10

100

TPC-H Query

Duration

[sec]

19/20





Q01 Q060.01

0.1

1

10

100

TPC-H Query

Duration

[sec]

19/20





Q01 Q060.01

0.1

1

10

100

TPC-H Query

Duration

[sec]

19/20

Conclusion

20/20

Conclusion

UDF support

20/20

Conclusion

UDF support

secondary index support

20/20

Conclusion

UDF support


build-in data structures

20/20

Conclusion

UDF support


bulk operations


20/20

Conclusion

UDF support


bulk operations


HSTORE

20/20

Conclusion

UDF support


bulk operations


HSTOREJSONB

20/20

An Experimental Analysis of Diﬀerent Key-Value Stores and … · 2017-03-08 · Key-Value Stores and Relational Databases David Gembalczyk Felix Martin Schuhknecht Jens Dittrich

Documents