Explaining the Postgres Query Optimizer BRUCE MOMJIAN January, 2012 The optimizer is the "brain" of the database, interpreting SQL queries and determining the fastest method of execution. This talk uses the EXPLAIN command to show how the optimizer interprets queries and determines optimal execution. Creative Commons Attribution License http://momjian.us/presentations 1 / 56
56
Embed
Explaining the Postgres Query Optimizer - PGCon 2014
The optimizer is the "brain" of the database, interpreting SQL queries and determining the fastest method of execution.
This talk uses the EXPLAIN command to show how the optimizer interprets queries and determines optimal execution. Examples include scan methods, index selection, join types, and how ANALYZE statistics influence their selection. The talk will assist developers and administrators in understanding how Postgres optimally executes their queries and what steps they can take to understand and perhaps improve its behavior.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Explaining the Postgres Query Optimizer
BRUCE MOMJIAN
January, 2012
The optimizer is the "brain" of the database, interpreting SQLqueries and determining the fastest method of execution. Thistalk uses the EXPLAIN command to show how the optimizerinterprets queries and determines optimal execution.Creative Commons Attribution License http://momjian.us/presentations
1 / 56
Postgres Query Execution
User
Terminal
CodeDatabase
Server
Application
Queries
Results
PostgreSQL
Libpq
Explaining the Postgres Query Optimizer 2 / 56
Postgres Query Execution
utility
Plan
Optimal Path
Query
Postmaster
Postgres Postgres
Libpq
Main
Generate Plan
Traffic Cop
Generate Paths
Execute Plan
e.g. CREATE TABLE, COPYSELECT, INSERT, UPDATE, DELETE
Rewrite Query
Parse Statement
UtilityCommand
Storage ManagersCatalogUtilities
Access Methods Nodes / Lists
Explaining the Postgres Query Optimizer 3 / 56
Postgres Query Execution
utility
Plan
Optimal Path
Query
Generate Plan
Traffic Cop
Generate Paths
Execute Plan
e.g. CREATE TABLE, COPYSELECT, INSERT, UPDATE, DELETE
WITH letter (letter, count) AS (SELECT letter, COUNT(*)FROM sampleGROUP BY 1
)SELECT letter AS l, count,
(SELECT *FROM lookup_letter(letter) AS l2LIMIT 1) AS lookup_letter
FROM letterORDER BY 2 DESC;
Explaining the Postgres Query Optimizer 24 / 56
Just the First EXPLAIN Lines
l | count | lookup_letter---+-------+-----------------------------------------------------------------------p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)t | 5 | Bitmap Heap Scan on sample (cost=4.29..12.76 rows=5 width=2)f | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)v | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)d | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)a | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)_ | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)u | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)(14 rows)
Explaining the Postgres Query Optimizer 25 / 56
We Can Force an Index Scan
SET enable_seqscan = false;
SET enable_bitmapscan = false;
WITH letter (letter, count) AS (SELECT letter, COUNT(*)FROM sampleGROUP BY 1
)SELECT letter AS l, count,
(SELECT *FROM lookup_letter(letter) AS l2LIMIT 1) AS lookup_letter
FROM letterORDER BY 2 DESC;
Explaining the Postgres Query Optimizer 26 / 56
Notice the High Cost for Common Values
l | count | lookup_letter---+-------+-------------------------------------------------------------------------p | 199 | Index Scan using i_sample on sample (cost=0.00..39.33 rows=199 width=2)s | 9 | Index Scan using i_sample on sample (cost=0.00..22.14 rows=9 width=2)c | 8 | Index Scan using i_sample on sample (cost=0.00..19.84 rows=8 width=2)r | 7 | Index Scan using i_sample on sample (cost=0.00..19.82 rows=7 width=2)t | 5 | Index Scan using i_sample on sample (cost=0.00..15.21 rows=5 width=2)d | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)v | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)f | 4 | Index Scan using i_sample on sample (cost=0.00..15.19 rows=4 width=2)_ | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)a | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)u | 3 | Index Scan using i_sample on sample (cost=0.00..12.88 rows=3 width=2)e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)(14 rows)RESET ALL;RESET
Explaining the Postgres Query Optimizer 27 / 56
This Was the Optimizer’s Preference
l | count | lookup_letter---+-------+-----------------------------------------------------------------------p | 199 | Seq Scan on sample (cost=0.00..13.16 rows=199 width=2)s | 9 | Seq Scan on sample (cost=0.00..13.16 rows=9 width=2)c | 8 | Seq Scan on sample (cost=0.00..13.16 rows=8 width=2)r | 7 | Seq Scan on sample (cost=0.00..13.16 rows=7 width=2)t | 5 | Bitmap Heap Scan on sample (cost=4.29..12.76 rows=5 width=2)f | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)v | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)d | 4 | Bitmap Heap Scan on sample (cost=4.28..12.74 rows=4 width=2)a | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)_ | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)u | 3 | Bitmap Heap Scan on sample (cost=4.27..11.38 rows=3 width=2)e | 2 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)i | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)k | 1 | Index Scan using i_sample on sample (cost=0.00..8.27 rows=1 width=2)(14 rows)
Explaining the Postgres Query Optimizer 28 / 56
Which Join Method?
◮ Nested Loop
◮ With Inner Sequential Scan◮ With Inner Index Scan
◮ Hash Join
◮ Merge Join
Explaining the Postgres Query Optimizer 29 / 56
What Is in pg_proc.oid?
SELECT oidFROM pg_procORDER BY 1LIMIT 8;oid-----3133343538394041
(8 rows)
Explaining the Postgres Query Optimizer 30 / 56
Create Temporary Tablesfrom pg_proc and pg_class
CREATE TEMPORARY TABLE sample1 (id, junk) ASSELECT oid, repeat(’x’, 250)FROM pg_procORDER BY random(); -- add rows in random order
SELECT 2256CREATE TEMPORARY TABLE sample2 (id, junk) AS
SELECT oid, repeat(’x’, 250)FROM pg_classORDER BY random(); -- add rows in random order
SELECT 260
These tables have no indexes and no optimizer statistics.
QUERY PLAN---------------------------------------------------------------------------------Nested Loop (cost=0.00..16.55 rows=1 width=254)-> Index Scan using i_sample1 on sample1 (cost=0.00..8.27 rows=1 width=4)
Index Cond: (id = 33::oid)-> Index Scan using i_sample2 on sample2 (cost=0.00..8.27 rows=1 width=258)
Index Cond: (sample2.id = 33::oid)(5 rows)
Explaining the Postgres Query Optimizer 47 / 56
Nested Loop Join with Inner Index Scan
aag
aar
aai
aay aag
aas
aar
aaa
aay
aai
aag
No Setup Required
Index Lookup
Index Must Already Exist
Outer Inner
Explaining the Postgres Query Optimizer 48 / 56
Pseudocode for Nested Loop Joinwith Inner Index Scan
for (i = 0; i < length(outer); i++)index_entry = get_first_match(outer[j])while (index_entry)
-> Index Scan using i_sample2 on sample2 (cost=0.00..52.15 rows=260 width=258)-> Index Scan using i_sample1 on sample1 (cost=0.00..1.62 rows=1 width=4)
Index Cond: (sample1.id = sample2.id)(5 rows)
Explaining the Postgres Query Optimizer 53 / 56
LIMIT 10
EXPLAIN SELECT sample2.id, sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)ORDER BY 1LIMIT 10;
-> Index Scan using i_sample2 on sample2 (cost=0.00..52.15 rows=260 width=258)-> Index Scan using i_sample1 on sample1 (cost=0.00..1.62 rows=1 width=4)
Index Cond: (sample1.id = sample2.id)(5 rows)
Explaining the Postgres Query Optimizer 54 / 56
LIMIT 100 Switches to Hash Join
EXPLAIN SELECT sample2.id, sample2.junkFROM sample1 JOIN sample2 ON (sample1.id = sample2.id)ORDER BY 1LIMIT 100;