Top Banner
DBs: harder , better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive
79

DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Apr 24, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

DBs: harder, better, faster, stronger reading

Maxime GosselinLead Software Architect

Oodrive

Page 2: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

The big plan

Page 3: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

The big planPostgreSQL… PostgreSQL… and the others

Page 4: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

The big planPostgreSQL… and the others

Storage

Page 5: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

The big planPostgreSQL… and the others

Storage

Indexes

Page 6: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

The big planPostgreSQL… and the others

Storage

Indexes

Algorithms, algorithms everywhere!

Page 7: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

The big planPostgreSQL… and the others

Storage

Indexes

Algorithms, algorithms everywhere!

Page 8: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Pages

Small, easy to read

Storage

Page 9: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Pages

Small, easy to read

Examples:

Storage

Postgresql, Oracle 8kbMySQL 16kb

Page 10: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Simple table, values:

● id: integer● value: text

Storage

Page 11: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Simple table, values:

● id: integer● value: text

Storage1, foo

2, bar

123, Oodrive

Page 12: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Sequential scanSELECT * FROM values WHERE id = 123;

Page 13: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Sequential scanSELECT * FROM values WHERE id = 123;

Page 14: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Sequential scanSELECT * FROM values WHERE id = 123;

Every page is read in order.

OK for small tables.

Scales in O(n)

Page 15: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Indexes

Page 16: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesIn the beginning was the binary search tree

Page 17: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesAt each node 10

8

1 9

20

15 123

Page 18: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesAt each node

● Left side smaller

10

8

1 9

20

15 123

Page 19: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesAt each node

● Left side lower● Right side higher

10

8

1 9

20

15 123

Page 20: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesAt each node

● Left side lower● Right side higher

Not good on disk

10

8

1 9

20

15 123

Page 21: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesLet’s complicate things

Page 22: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesLet’s complicate things

Here comes the B-tree

Page 23: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesLet’s complicate things

Here comes the B-tree

(Nobody knows what the B means…)

Page 24: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesB-tree 7 16

9 12 151 2 6 2118

Index keys

Page 25: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesB-tree

● Internal nodes:Linked to other nodes

7 16

9 12 151 2 6 2118

Page 26: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesB-tree

● Internal nodes:linked to other nodes

● Leaves: linked to data pages

7 16

9 12 151 2 6 2118

Page 27: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesB-tree

● Internal order

7 16

9 12 151 2 6 2118

<

< < < < <

Page 28: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesB-tree

● Internal order● Left side lower

7 16

9 12 151 2 6 2118

Page 29: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesB-tree

● Internal order● Left side lower● Right side higher

7 16

9 12 151 2 6 2118

Page 30: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

IndexesB-tree

Fits well on a page=> Efficient on disk

7 16

9 12 151 2 6 2118

Page 31: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What can we do with B-trees?Finding a single value

Let’s look for 12

7 16

9 12 151 2 6 2118

Page 32: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What can we do with B-trees?7 < 12 < 16 7 16

9 12 151 2 6 2118

Page 33: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What can we do with B-trees?7 < 12 < 16

=> follow the pointer

7 16

9 12 151 2 6 2118

Page 34: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What can we do with B-trees?7 < 12 < 16

=> follow the pointer

Find it in the page

7 16

9 12 151 2 6 2118

Page 35: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What can we do with B-trees?7 < 12 < 16

=> follow the pointer

Find it in the page

Complexity O(log(n))

7 16

9 12 151 2 6 2118

Page 36: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What can we do with B-trees?Walk the tree (or subtree) in order

Simple depth firstsearch

7 16

9 12 151 2 6 2118

Page 37: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What can we do with B-trees?Walk the tree (or subtree) in order

=> Answer ORDER BY queries

7 16

9 12 151 2 6 2118

Page 38: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What CAN’T we do with B-trees?Answer queries with:

● AND clauses using multiple trees

Page 39: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What CAN’T we do with B-trees?Answer queries with:

● AND clauses using multiple trees

SELECT * FROM table WHERE col1 = 123 AND col2 = ‘abc’;

With an index on col1 and another on col2.

Page 40: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What CAN’T we do with B-trees?Answer queries with:

● AND clauses using multiple trees

SELECT * FROM table WHERE col1 = 123 AND col2 = ‘abc’;

With an index on col1 and another on col2.

But a single index on (col1, col2) would work nicely.

Page 41: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What CAN’T we do with B-trees?Answer queries with:

● clauses on values derived from an indexed column

SELECT * FROM people WHERE lower(name) = ‘maxime’;

Page 42: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

What CAN’T we do with B-trees?Answer queries with:

● clauses on values derived from an indexed column

SELECT * FROM people WHERE lower(name) = ‘maxime’;

But you can build an index on (lower(name))

Page 43: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Multi column indexesColumn order matters!

(Company, city)

Blablacar, ParisCanal+, Issy les MoulineauxOodrive, LyonOodrive, Paris...

Page 44: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Multi column indexesColumn order matters!

(Company, city)

Blablacar, ParisCanal+, Issy les MoulineauxOodrive, LyonOodrive, Paris...

Graphs are too hard.But it’s still a B-tree!

Page 45: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Multi column indexesColumn order matters!

(Company, city)

It’s easy to find all the cities for Oodrive

Blablacar, ParisCanal+, Issy les MoulineauxOodrive, LyonOodrive, Paris...

Page 46: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Multi column indexesColumn order matters!

(Company, city)

It’s hard to find who works in Paris

Blablacar, ParisCanal+, Issy les MoulineauxOodrive, LyonOodrive, Paris...

Page 47: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Multi column indexesColumn order matters!

(City, Company)

It’s easy to find who works in Paris

Issy les Moulineaux, Canal+Lyon, OodriveParis, BlablacarParis, Oodrive...

Page 48: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Multi column indexesColumn order matters!

(City, Company)

It’s hard to find all the cities for Oodrive

Issy les Moulineaux, Canal+Lyon, OodriveParis, BlablacarParis, Oodrive...

Page 49: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables

Page 50: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tablesSELECT * FROM table1

JOIN table2 USING (joinColumn1)

WHERE ...;

Page 51: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: nested loopStandard developper approach:

for(int i = 0; i < table1.length; ++i) {filter(table1[i]); ….for(int j = 0; j < table2.length; ++j) {

filter(table2[j]); ….}

}

Page 52: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: nested loopWorks well if:

● the first table is small,● the second table has an index on the join column.

Page 53: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: nested loopWorks well if:

● the first table is small,● the second table has an index on the join column.

Doesn’t if:

● the join condition is complex

Page 54: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: merge joinSort the two sides of the join on the index condition

Iterate both sides to find common values.

Page 55: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: merge join1210121520

131520

Page 56: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: merge join1210121520

131520

Page 57: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: merge join1210121520

131520

Page 58: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: merge joinWorks well if:

● both sides can be filtered and sorted cheaply● both sides are big

Best strategy for full outer joins

Page 59: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: hash joinOn the smaller side:

● filter the data● put each row in a hash table, indexed by the join columns

Page 60: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: hash joinSELECT * FROM t1

JOIN t2 using(a, b);

Row1: a1, b1, c1, ...

Row2: a2, b2, c2, ...

Page 61: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: hash joinSELECT * FROM t1

JOIN t2 using(a, b);

row1: a1, b1, c1, ...

row2: a2, b2, c2, ...

hash(a1, b1) => row1, rowNhash(a2, b2) => row2….

Page 62: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: hash joinOn the smaller side:

● filter the data● put each row in a hash table, indexed by the join columns

On the larger side

● filter the data● for every row, retrieve the smaller side from the hash table

Page 63: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: hash joinSELECT * FROM t1

JOIN t2 using(a, b);

row1: a1, b1, c1, ...

row2: a2, b2, c2, ...

hash(a1, b1) => row1, rowNhash(a2, b2) => row2….

Page 64: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Joining tables: hash joinAdvantages:

● can work without indexes on the join columns● single traversal of the smaller side

Drawbacks

● the hash table should fit in memory

Page 65: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Planning & optimizing

Now what??SELECT * FROM table1 JOIN table2JOIN table3 ….. JOIN tableN ….WHERE sky = ‘blue’ AND …AND condN

Page 66: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Planning & optimizingGoal: least possible work

=> read as little as possible

Page 67: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Planning & optimizingThe tools:

● a query tree that can be modified● cardinality estimation for column values

Page 68: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Planning & optimizingHow can it be used?

● generate all possible execution plans(what index to use, join types and order...)

● compute the costs of each plan

Page 69: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Planning & optimizingHow can it be used?

● generate all possible execution plans(what index to use, join types and order...)

● compute the costs of each plan

In the end, there can be only one.

Page 70: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Planning & optimizing

Hash Join (cost=14.25..246593.00 rows=1338306 width=15) Hash Cond: (series.val = even.val) -> Seq Scan on series (cost=0.00..186311.50 rows=12502450 width=12) -> Hash (cost=8.00..8.00 rows=500 width=7) -> Seq Scan on even (cost=0.00..8.00 rows=500 width=7)

Page 71: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Planning & optimizing

Hash Join (cost=14.25..246593.00 rows=1338306 width=15) Hash Cond: (series.val = even.val) -> Seq Scan on series (cost=0.00..186311.50 rows=12502450 width=12) -> Hash (cost=8.00..8.00 rows=500 width=7) -> Seq Scan on even (cost=0.00..8.00 rows=500 width=7)

Startup cost Total cost Rows expected

Page 72: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

CaveatsLet’s index everything!

Page 73: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

CaveatsLet’s index everything!

Index only what you need, they are costly

Page 74: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

CaveatsLet’s index everything!

Index only what you need, indexes are costly

Small dataset, small gains

Page 75: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Other interesting things● data modification & MVCC

(Multiversion concurrency control)

How are transactions represented?How do we keep uncommitted writes from being read?

Page 76: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Other interesting things● data modification & MVCC

(Multiversion concurrency control)● replication

What are the tradeoffs between consistency and availability?

Page 77: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Other interesting things● data modification & MVCC

(Multiversion concurrency control)● replication● other index types

Inverted indexes, bitmap, hash, spatial...

Page 78: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

The end

Page 79: DBs: harder, better, faster, Oodrive stronger reading Lead ......DBs: harder, better, faster, stronger reading Maxime Gosselin Lead Software Architect Oodrive

Oh, you had questions?