Top Banner
Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, interpreting SQL queries and determining the fastest method of execution. This talk uses the EXPLAIN command to show how the optimizer interprets queries and determines optimal execution. Creative Commons Attribution License http://momjian.us/presentations Last updated: September, 2018 1 / 56
56

Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Dec 23, 2018

Download

Documents

hatu
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Explaining the Postgres Query Optimizer

BRUCE MOMJIAN

The optimizer is the "brain" of the database, interpreting SQLqueries and determining the fastest method of execution. Thistalk uses the EXPLAIN command to show how the optimizerinterprets queries and determines optimal execution.Creative Commons Attribution License http://momjian.us/presentations

Last updated: September, 2018

1 / 56

Page 2: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Postgres Query Execution

User

Terminal

CodeDatabase

Server

Application

Queries

Results

PostgreSQL

Libpq

2 / 56

Page 3: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Postgres Query Execution

utility

Plan

Optimal Path

Query

Postmaster

Postgres Postgres

Libpq

Main

Generate Plan

Traffic Cop

Generate Paths

Execute Plan

e.g. CREATE TABLE, COPYSELECT, INSERT, UPDATE, DELETE

Rewrite Query

Parse Statement

UtilityCommand

Storage ManagersCatalogUtilities

Access Methods Nodes / Lists

3 / 56

Page 4: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Postgres Query Execution

utility

Plan

Optimal Path

Query

Generate Plan

Traffic Cop

Generate Paths

Execute Plan

e.g. CREATE TABLE, COPYSELECT, INSERT, UPDATE, DELETE

Rewrite Query

Parse Statement

UtilityCommand

4 / 56

Page 5: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

The Optimizer Is the Brain

https://www.flickr.com/photos/dierkschaefer/

5 / 56

Page 6: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

What Decisions Does the Optimizer Have to Make?

◮ Scan Method

◮ Join Method

◮ Join Order

6 / 56

Page 7: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Which Scan Method?

◮ Sequential Scan

◮ Bitmap Index Scan

◮ Index Scan

7 / 56

Page 8: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

A Simple Example Using pg_class.relname

SELECT relnameFROM pg_classORDER BY 1LIMIT 8;

relname-----------------------------------_pg_foreign_data_wrappers_pg_foreign_servers_pg_user_mappingsadministrable_role_authorizationsapplicable_rolesattributescheck_constraint_routine_usagecheck_constraints

8 / 56

Page 9: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Let’s Use Just the First Letter of pg_class.relname

SELECT substring(relname, 1, 1)FROM pg_classORDER BY 1LIMIT 8;substring-----------___aaacc

9 / 56

Page 10: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Create a Temporary Table with an Index

CREATE TEMPORARY TABLE sample (letter, junk) ASSELECT substring(relname, 1, 1), repeat(’x’, 250)FROM pg_classORDER BY random(); -- add rows in random order

CREATE INDEX i_sample on sample (letter);

All queries used in this presentation are available at http://momjian.us/main/writings/pgsql/optimizer.sql.

10 / 56

http://momjian.us/main/writings/pgsql/optimizer.sql
http://momjian.us/main/writings/pgsql/optimizer.sql
Page 11: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Create an EXPLAIN Function

CREATE OR REPLACE FUNCTION lookup_letter(text) RETURNS SETOF text AS $$BEGINRETURN QUERY EXECUTE ’

EXPLAIN SELECT letterFROM sampleWHERE letter = ’’’ || $1 || ’’’’;

END$$ LANGUAGE plpgsql;

11 / 56

Page 12: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

What is the Distribution of the sample Table?

WITH letters (letter, count) AS (SELECT letter, COUNT(*)FROM sampleGROUP BY 1

)SELECT letter, count, (count * 100.0 / (SUM(count) OVER ()))::numeric(4,1) AS "%"FROM lettersORDER BY 2 DESC;

12 / 56

Page 13: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

What is the Distribution of the sample Table?

letter | count | %--------+-------+------p | 199 | 78.7s | 9 | 3.6c | 8 | 3.2r | 7 | 2.8t | 5 | 2.0v | 4 | 1.6f | 4 | 1.6d | 4 | 1.6u | 3 | 1.2a | 3 | 1.2_ | 3 | 1.2e | 2 | 0.8i | 1 | 0.4k | 1 | 0.4

13 / 56

Page 14: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Is the Distribution Important?

EXPLAIN SELECT letterFROM sampleWHERE letter = ’p’;

QUERY PLAN------------------------------------------------------------------------Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=32)Index Cond: (letter = ’p’::text)

14 / 56

Page 15: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Is the Distribution Important?

EXPLAIN SELECT letterFROM sampleWHERE letter = ’d’;

QUERY PLAN------------------------------------------------------------------------Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=32)Index Cond: (letter = ’d’::text)

15 / 56

Page 16: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Is the Distribution Important?

EXPLAIN SELECT letterFROM sampleWHERE letter = ’k’;

QUERY PLAN------------------------------------------------------------------------Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=32)Index Cond: (letter = ’k’::text)

16 / 56

Page 17: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Running ANALYZE Causesa Sequential Scan for a Common Value

ANALYZE sample;

EXPLAIN SELECT letterFROM sampleWHERE letter = ’p’;

QUERY PLAN---------------------------------------------------------Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)Filter: (letter = ’p’::text)

Autovacuum cannot ANALYZE (or VACUUM) temporary tables becausethese tables are only visible to the creating session.

17 / 56

Page 18: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Sequential Scan

TA

DATA

DATA

DATA

DATA

DATA

DATA

DATA

DATA

D

8K

Heap

A

A

D

TATA

DATA

DA

18 / 56

Page 19: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

A Less Common Value Causes a Bitmap Index Scan

EXPLAIN SELECT letterFROM sampleWHERE letter = ’d’;

QUERY PLAN-----------------------------------------------------------------------Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)Recheck Cond: (letter = ’d’::text)-> Bitmap Index Scan on i_sample (cost=0.00..4.28 rows=4 width=0)

Index Cond: (letter = ’d’::text)

19 / 56

Page 20: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Bitmap Index Scan

=&

Combined

’A’ AND ’NS’

1

0

1

0

TableIndex 1

col1 = ’A’

Index 2

1

0

0

col2 = ’NS’

1 0

1

0

0

Index

20 / 56

Page 21: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

An Even Rarer Value Causes an Index Scan

EXPLAIN SELECT letterFROM sampleWHERE letter = ’k’;

QUERY PLAN-----------------------------------------------------------------------Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)Index Cond: (letter = ’k’::text)

21 / 56

Page 22: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Index Scan

A

DATA

DATA

DATA

DATA

DATA

DATA

D

< >=Key

< >=Key

Index

Heap

< >=Key

ATA

DATA

DATA

DATA

DATA

DAT

22 / 56

Page 23: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Let’s Look at All Values and their Effects

WITH letter (letter, count) AS (SELECT letter, COUNT(*)FROM sampleGROUP BY 1

)SELECT letter AS l, count, lookup_letter(letter)FROM letterORDER BY 2 DESC;

l | count | lookup_letter---+-------+-----------------------------------------------------------------------p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)p | 199 | Filter: (letter = ’p’::text)s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)s | 9 | Filter: (letter = ’s’::text)c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)c | 8 | Filter: (letter = ’c’::text)r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)r | 7 | Filter: (letter = ’r’::text)

…

23 / 56

Page 24: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

OK, Just the First Lines

WITH letter (letter, count) AS (SELECT letter, COUNT(*)FROM sampleGROUP BY 1

)SELECT letter AS l, count,

(SELECT *FROM lookup_letter(letter) AS l2LIMIT 1) AS lookup_letter

FROM letterORDER BY 2 DESC;

24 / 56

Page 25: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Just the First EXPLAIN Lines

l | count | lookup_letter---+-------+-----------------------------------------------------------------------p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)t | 5 | Bitmap Heap Scan on sample (cost=4.29..12.76 rows=5 width=2)f | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)v | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)d | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)a | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)_ | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)u | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)

25 / 56

Page 26: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

We Can Force an Index Scan

SET enable_seqscan = false;

SET enable_bitmapscan = false;

WITH letter (letter, count) AS (SELECT letter, COUNT(*)FROM sampleGROUP BY 1

)SELECT letter AS l, count,

(SELECT *FROM lookup_letter(letter) AS l2LIMIT 1) AS lookup_letter

FROM letterORDER BY 2 DESC;

26 / 56

Page 27: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Notice the High Cost for Common Values

l | count | lookup_letter---+-------+-------------------------------------------------------------------------p | 199 | Index Scan using i_sample on sample (cost=0.00..39.33 rows=199 width=2)s | 9 | Index Scan using i_sample on sample (cost=0.00..22.14 rows=9 width=2)c | 8 | Index Scan using i_sample on sample (cost=0.00..19.84 rows=8 width=2)r | 7 | Index Scan using i_sample on sample (cost=0.00..19.82 rows=7 width=2)t | 5 | Index Scan using i_sample on sample (cost=0.00..15.21 rows=5 width=2)d | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)v | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)f | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)_ | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)a | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)u | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)

RESET ALL;

27 / 56

Page 28: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

This Was the Optimizer’s Preference

l | count | lookup_letter---+-------+-----------------------------------------------------------------------p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)t | 5 | Bitmap Heap Scan on sample (cost=4.29..12.76 rows=5 width=2)f | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)v | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)d | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)a | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)_ | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)u | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)

28 / 56

Page 29: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Which Join Method?

◮ Nested Loop

◮ With Inner Sequential Scan◮ With Inner Index Scan

◮ Hash Join

◮ Merge Join

29 / 56

Page 30: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

What Is in pg_proc.oid?

SELECT oidFROM pg_procORDER BY 1LIMIT 8;oid-----3133343538394041

30 / 56

Page 31: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Create Temporary Tablesfrom pg_proc and pg_class

CREATE TEMPORARY TABLE sample1 (id, junk) ASSELECT oid, repeat(’x’, 250)FROM pg_procORDER BY random(); -- add rows in random order

CREATE TEMPORARY TABLE sample2 (id, junk) ASSELECT oid, repeat(’x’, 250)FROM pg_classORDER BY random(); -- add rows in random order

These tables have no indexes and no optimizer statistics.

31 / 56

Page 32: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Join the Two Tableswith a Tight Restriction

EXPLAIN SELECT sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)WHERE sample1.id = 33;

QUERY PLAN---------------------------------------------------------------------Nested Loop (cost=0.00..234.68 rows=300 width=32)

-> Seq Scan on sample1 (cost=0.00..205.54 rows=50 width=4)Filter: (id = 33::oid)

-> Materialize (cost=0.00..25.41 rows=6 width=36)-> Seq Scan on sample2 (cost=0.00..25.38 rows=6 width=36)

Filter: (id = 33::oid)

32 / 56

Page 33: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Nested Loop Joinwith Inner Sequential Scan

aag

aar

aay aag

aas

aar

aaa

aay

aai

aag

No Setup Required

aai

Used For Small Tables

Outer Inner

33 / 56

Page 34: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Pseudocode for Nested Loop Joinwith Inner Sequential Scan

for (i = 0; i < length(outer); i++)for (j = 0; j < length(inner); j++)

if (outer[i] == inner[j])output(outer[i], inner[j]);

34 / 56

Page 35: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Join the Two Tables with a Looser Restriction

EXPLAIN SELECT sample1.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)WHERE sample2.id > 33;

QUERY PLAN----------------------------------------------------------------------Hash Join (cost=30.50..950.88 rows=20424 width=32)

Hash Cond: (sample1.id = sample2.id)-> Seq Scan on sample1 (cost=0.00..180.63 rows=9963 width=36)-> Hash (cost=25.38..25.38 rows=410 width=4)

-> Seq Scan on sample2 (cost=0.00..25.38 rows=410 width=4)Filter: (id > 33::oid)

35 / 56

Page 36: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Hash Join

Hashed

Must fit in Main Memory

aak

aar

aak

aay aaraam

aao aaw

aay

aag

aas

Outer Inner

36 / 56

Page 37: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Pseudocode for Hash Join

for (j = 0; j < length(inner); j++)hash_key = hash(inner[j]);append(hash_store[hash_key], inner[j]);

for (i = 0; i < length(outer); i++)hash_key = hash(outer[i]);for (j = 0; j < length(hash_store[hash_key]); j++)

if (outer[i] == hash_store[hash_key][j])output(outer[i], inner[j]);

37 / 56

Page 38: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Join the Two Tables with No Restriction

EXPLAIN SELECT sample1.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id);

QUERY PLAN-------------------------------------------------------------------------Merge Join (cost=927.72..1852.95 rows=61272 width=32)

Merge Cond: (sample2.id = sample1.id)-> Sort (cost=85.43..88.50 rows=1230 width=4)

Sort Key: sample2.id-> Seq Scan on sample2 (cost=0.00..22.30 rows=1230 width=4)

-> Sort (cost=842.29..867.20 rows=9963 width=36)Sort Key: sample1.id-> Seq Scan on sample1 (cost=0.00..180.63 rows=9963 width=36)

38 / 56

Page 39: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Merge Join

Sorted

Sorted

Ideal for Large Tables

An Index Can Be Used to Eliminate the Sort

aaa

aab

aac

aad

aaa

aab

aab

aaf

aaf

aac

aae

Outer Inner

39 / 56

Page 40: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Pseudocode for Merge Join

sort(outer);sort(inner);i = 0;j = 0;save_j = 0;while (i < length(outer))if (outer[i] == inner[j])

output(outer[i], inner[j]);if (outer[i] <= inner[j] && j < length(inner))

j++;if (outer[i] < inner[j])

save_j = j;else

i++;j = save_j;

40 / 56

Page 41: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Order of Joined Relations Is Insignificant

EXPLAIN SELECT sample2.junkFROM sample2 JOIN sample1 ON (sample2.id = sample1.id);

QUERY PLAN------------------------------------------------------------------------Merge Join (cost=927.72..1852.95 rows=61272 width=32)

Merge Cond: (sample2.id = sample1.id)-> Sort (cost=85.43..88.50 rows=1230 width=36)

Sort Key: sample2.id-> Seq Scan on sample2 (cost=0.00..22.30 rows=1230 width=36)

-> Sort (cost=842.29..867.20 rows=9963 width=4)Sort Key: sample1.id-> Seq Scan on sample1 (cost=0.00..180.63 rows=9963 width=4)

The most restrictive relation, e.g., sample2, is always on the outer side ofmerge joins. All previous merge joins also had sample2 in outer position.

41 / 56

Page 42: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Add Optimizer Statistics

ANALYZE sample1;

ANALYZE sample2;

42 / 56

Page 43: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

This Was a Merge Join without Optimizer Statistics

EXPLAIN SELECT sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id);

QUERY PLAN------------------------------------------------------------------------Hash Join (cost=15.85..130.47 rows=260 width=254)

Hash Cond: (sample1.id = sample2.id)-> Seq Scan on sample1 (cost=0.00..103.56 rows=2256 width=4)-> Hash (cost=12.60..12.60 rows=260 width=258)

-> Seq Scan on sample2 (cost=0.00..12.60 rows=260 width=258)

43 / 56

Page 44: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Outer Joins Can Affect Optimizer Join Usage

EXPLAIN SELECT sample1.junkFROM sample1 RIGHT OUTER JOIN sample2 ON (sample1.id = sample2.id);

QUERY PLAN--------------------------------------------------------------------------Hash Left Join (cost=131.76..148.26 rows=260 width=254)

Hash Cond: (sample2.id = sample1.id)-> Seq Scan on sample2 (cost=0.00..12.60 rows=260 width=4)-> Hash (cost=103.56..103.56 rows=2256 width=258)

-> Seq Scan on sample1 (cost=0.00..103.56 rows=2256 width=258)

44 / 56

Page 45: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Cross Joins Are Nested Loop Joinswithout Join Restriction

EXPLAIN SELECT sample1.junkFROM sample1 CROSS JOIN sample2;

QUERY PLAN----------------------------------------------------------------------Nested Loop (cost=0.00..7448.81 rows=586560 width=254)

-> Seq Scan on sample1 (cost=0.00..103.56 rows=2256 width=254)-> Materialize (cost=0.00..13.90 rows=260 width=0)

-> Seq Scan on sample2 (cost=0.00..12.60 rows=260 width=0)

45 / 56

Page 46: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Create Indexes

CREATE INDEX i_sample1 on sample1 (id);

CREATE INDEX i_sample2 on sample2 (id);

46 / 56

Page 47: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Nested Loop with Inner Index Scan Now Possible

EXPLAIN SELECT sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)WHERE sample1.id = 33;

QUERY PLAN---------------------------------------------------------------------------------Nested Loop (cost=0.00..16.55 rows=1 width=254)-> Index Scan using i_sample1 on sample1 (cost=0.00..8.27 rows=1 width=4)

Index Cond: (id = 33::oid)-> Index Scan using i_sample2 on sample2 (cost=0.00..8.27 rows=1 width=258)

Index Cond: (sample2.id = 33::oid)

47 / 56

Page 48: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Nested Loop Join with Inner Index Scan

aag

aar

aai

aay aag

aas

aar

aaa

aay

aai

aag

No Setup Required

Index Lookup

Index Must Already Exist

Outer Inner

48 / 56

Page 49: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Pseudocode for Nested Loop Joinwith Inner Index Scan

for (i = 0; i < length(outer); i++)index_entry = get_first_match(outer[j])while (index_entry)

output(outer[i], inner[index_entry]);index_entry = get_next_match(index_entry);

49 / 56

Page 50: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Query Restrictions Affect Join Usage

EXPLAIN SELECT sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)WHERE sample2.junk ˜ ’^aaa’;

QUERY PLAN-------------------------------------------------------------------------------Nested Loop (cost=0.00..21.53 rows=1 width=254)-> Seq Scan on sample2 (cost=0.00..13.25 rows=1 width=258)

Filter: (junk ˜ ’^aaa’::text)-> Index Scan using i_sample1 on sample1 (cost=0.00..8.27 rows=1 width=4)

Index Cond: (sample1.id = sample2.id)

No junk rows begin with ’aaa’.

50 / 56

Page 51: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

All ’junk’ Columns Begin with ’xxx’

EXPLAIN SELECT sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)WHERE sample2.junk ˜ ’^xxx’;

QUERY PLAN------------------------------------------------------------------------Hash Join (cost=16.50..131.12 rows=260 width=254)Hash Cond: (sample1.id = sample2.id)-> Seq Scan on sample1 (cost=0.00..103.56 rows=2256 width=4)-> Hash (cost=13.25..13.25 rows=260 width=258)

-> Seq Scan on sample2 (cost=0.00..13.25 rows=260 width=258)Filter: (junk ˜ ’^xxx’::text)

Hash join was chosen because many more rows are expected. Thesmaller table, e.g., sample2, is always hashed.

51 / 56

Page 52: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Without LIMIT, Hash Is Usedfor this Unrestricted Join

EXPLAIN SELECT sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id);

QUERY PLAN------------------------------------------------------------------------Hash Join (cost=15.85..130.47 rows=260 width=254)

Hash Cond: (sample1.id = sample2.id)-> Seq Scan on sample1 (cost=0.00..103.56 rows=2256 width=4)-> Hash (cost=12.60..12.60 rows=260 width=258)

-> Seq Scan on sample2 (cost=0.00..12.60 rows=260 width=258)

52 / 56

Page 53: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

LIMIT Can Affect Join Usage

EXPLAIN SELECT sample2.id, sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)ORDER BY 1LIMIT 1;

QUERY PLAN------------------------------------------------------------------------------------------Limit (cost=0.00..1.83 rows=1 width=258)-> Nested Loop (cost=0.00..477.02 rows=260 width=258)

-> Index Scan using i_sample2 on sample2 (cost=0.00..52.15 rows=260 width=258)-> Index Scan using i_sample1 on sample1 (cost=0.00..1.62 rows=1 width=4)

Index Cond: (sample1.id = sample2.id)

53 / 56

Page 54: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

LIMIT 10

EXPLAIN SELECT sample2.id, sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)ORDER BY 1LIMIT 10;

QUERY PLAN------------------------------------------------------------------------------------------Limit (cost=0.00..18.35 rows=10 width=258)-> Nested Loop (cost=0.00..477.02 rows=260 width=258)

-> Index Scan using i_sample2 on sample2 (cost=0.00..52.15 rows=260 width=258)-> Index Scan using i_sample1 on sample1 (cost=0.00..1.62 rows=1 width=4)

Index Cond: (sample1.id = sample2.id)

54 / 56

Page 55: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

LIMIT 100 Switches to Hash Join

EXPLAIN SELECT sample2.id, sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)ORDER BY 1LIMIT 100;

QUERY PLAN------------------------------------------------------------------------------------Limit (cost=140.41..140.66 rows=100 width=258)-> Sort (cost=140.41..141.06 rows=260 width=258)

Sort Key: sample2.id-> Hash Join (cost=15.85..130.47 rows=260 width=258)

Hash Cond: (sample1.id = sample2.id)-> Seq Scan on sample1 (cost=0.00..103.56 rows=2256 width=4)-> Hash (cost=12.60..12.60 rows=260 width=258)

-> Seq Scan on sample2 (cost=0.00..12.60 rows=260 width=258)

55 / 56

Page 56: Explaining the Postgres Query Optimizer - Bruce Momjian · Explaining the Postgres Query Optimizer BRUCE MOMJIAN The optimizer is the "brain" of the database, ... PostgreSQL Libpq

Conclusion

http://momjian.us/presentations https://www.flickr.com/photos/trevorklatko/

56 / 56


Related Documents
The Magic of Hot Streaming Replication, Bruce Momjian

The Magic of Hot Streaming Replication, Bruce Momjian

Category: Technology
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)

Non-Relational Postgres / Bruce Momjian (EnterpriseDB)

Category: Engineering
Explaining the Postgres Query Optimizer (Bruce Momjian)

Explaining the Postgres Query Optimizer (Bruce Momjian)

Category: Documents
Processing Data Inside PostgreSQL - Bruce Momjian

Processing Data Inside PostgreSQL - Bruce Momjian

Category: Documents
Unlocking the Postgres Lock Manager - Momjian · Unlocking the Postgres Lock Manager BRUCE MOMJIAN This talk explores all aspects of locking in Postgres by showing queries and their

Unlocking the Postgres Lock Manager - Momjian · Unlocking....

Category: Documents
Using VMware vFabric Postgres - vFabric Postgres 9.3.5

Using VMware vFabric Postgres - vFabric Postgres 9.3.5

Category: Documents
The Future of Postgres Sharding - Bruce Momjian · The Future of Postgres Sharding BRUCE MOMJIAN This presentation will cover the advantages of sharding and future Postgres sharding

The Future of Postgres Sharding - Bruce Momjian · The...

Category: Documents
Koichi Suzuki - Postgres-XC Dynamic Cluster Management @ Postgres Open

Koichi Suzuki - Postgres-XC Dynamic Cluster Management @...

Category: Technology
Managing Global Teams - Bruce Momjian · Managing Global Teams BRUCE MOMJIAN This presentation highlights the challenges of motivating and managing a global team of employees or volunteers.

Managing Global Teams - Bruce Momjian · Managing Global...

Category: Documents
How the Postgres Query Optimizer Works

How the Postgres Query Optimizer Works

Category: Technology
EDB Postgres Advanced Server Installation Guide · EDB Postgres™ Advanced Server Installation Guide EDB Postgres™ Advanced Server 9.5 formerly Postgres Plus Advanced Server 9.5

EDB Postgres Advanced Server Installation Guide · EDB...

Category: Documents
Inside PostgreSQL Shared Memory - Momjian · Inside PostgreSQL Shared Memory BRUCE MOMJIAN POSTGRESQL is an open-source, full-featured relational database. ... " jne 1f \n" ...

Inside PostgreSQL Shared Memory - Momjian · Inside...

Category: Documents