MySQL Query Optimization

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Query Optimization

Morgan Tocker MySQL Community Manager October, 2014


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |4

Today’s Agenda

Introduction

World Schema

IMDB Schema

Advanced Topics

1

2

3

4


Introduction

• SQL is declarative. • You say what you want, not how to process. • Can’t sight check a query to understand how it executes. • Database management chooses the best possible way -‐ think like a GPS navigator.

5


GPS Software

6


MySQL Optimizer

7


Diagnostic Commands

• EXPLAIN (all versions) • EXPLAIN FORMAT=JSON (MySQL 5.6+) • Workbench supports visual format.

• OPTIMIZER TRACE (MySQL 5.6+)

8


Today’s Agenda

Introduction

World Schema

IMDB Schema

Advanced Topics

1

2

3

4

2

1


The World Schema

• Contains cities, countries and language statistics. • Download From: • http://dev.mysql.com/doc/index-‐other.html

• Very small data set -‐ good for learning. • Not perfect for explaining performance differences

10

http://dev.mysql.com/doc/index-other.html


Find Countries in Asia with Population >5M

11

mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 239 filtered: 4.76 Extra: Using where1 row in set, 1 warning (0.00 sec)

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 12

EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "53.80" }, "table": { "table_name": "Country", "access_type": "ALL", "rows_examined_per_scan": 239, "rows_produced_per_join": 11, "filtered": 4.7614, "cost_info": { "read_cost": "51.52", "eval_cost": "2.28", "prefix_cost": "53.80", "data_read_per_join": "2K" }, "used_columns": [ "Code", .. "Capital", "Code2" ], "attached_condition": "((`world`.`country`.`Continent` = 'Asia') and (`world`.`country`.`Population` > 5000000))" } } }

FORMAT=JSONAvailable since MySQL 5.6. Expanded in 5.7 to add cost information.


What indexes will make this query faster?

• Some suggestions: • Index on (population) • Index on (continent) • Index on (population, continent) • Index on (continent, population)

13

SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000;


Next Question

• Given so many choices, which is the best choice? • A good GPS navigator understands traffic and finds the fastest route. • A good query optimizer does similar.

14


Query Optimizer Strategy

• Pick the plan that consumes the least amount of resources (CPU, IO).

15


Query Optimizer Strategy (cont.)

• Use statistics available to identify good and bad plans. • Decisions may include: • Which indexes are good/bad. • Which order to join tables in. • If indexes should be read in a particular order to avoid sorting, or if sorting is not expensive.

16


Using single column indexes first….

• p (population) • c (continent)

17


mysql> ALTER TABLE Country ADD INDEX p (population);Query OK, 0 rows affected (0.04 sec)Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G************************ 1. row ************************ id: 1 select_type: SIMPLE table: Country partitions: NULL type: ALLpossible_keys: p key: NULL key_len: NULL ref: NULL rows: 239 filtered: 6.46 Extra: Using where1 row in set, 1 warning (0.00 sec)


Why would an index not be used?

19

mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country FORCE INDEX (p) WHERE Continent = 'Asia' AND population > 5000000\G******* 1. row *******EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "152.21" }, "table": { "table_name": "Country", "access_type": "range", "possible_keys": [ "p" ],

mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G******* 1. row *******EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "53.80" }, "table": { "table_name": "Country", "access_type": "ALL", "possible_keys": [ "p" ],

Using MySQL 5.7+


Modifying the query range: >50M

20

mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: p key: p key_len: 4 ref: NULL rows: 24 filtered: 14.29 Extra: Using index condition; Using where1 row in set, 1 warning (0.00 sec)


mysql> ALTER TABLE Country ADD INDEX c (continent);Query OK, 0 rows affected (0.04 sec)Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G*********************** 1. row *********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: refpossible_keys: p,c key: c key_len: 1 ref: const rows: 51 filtered: 45.19 Extra: Using where1 row in set, 1 warning (0.00 sec)


mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 500000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: p,c key: p key_len: 4 ref: NULL rows: 2 filtered: 21.34 Extra: Using index condition; Using where1 row in set, 1 warning (0.00 sec)

Modifying the query range: >500M


Composite Indexes

• On >50M which is better? • (population, continent) • (continent, population)

23


mysql> ALTER TABLE Country ADD INDEX pc (pop..on, co..nt);Query OK, 0 rows affected (0.02 sec)Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: refpossible_keys: p,c,pc key: c key_len: 1 ref: const rows: 51 filtered: 10.04 Extra: Using where1 row in set, 1 warning (0.00 sec)


mysql> EXPLAIN SELECT * FROM Country FORCE INDEX (pc) WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: pc key: pc key_len: 4 ref: NULL rows: 24 filtered: 14.29 Extra: Using index condition1 row in set, 1 warning (0.00 sec)

FORCE INDEX


mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country FORCE INDEX (pc) WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row **********************EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "34.61" }, "table": {.. ], "key": "pc", "used_key_parts": [ "Population" ],..

Easier to see with JSON EXPLAIN


Rule of Thumb

• Index on (const, range) instead of (range, const). • Applies to all databases.

27


mysql> ALTER TABLE Country ADD INDEX cp (co..nt, pop..on);Query OK, 0 rows affected (0.02 sec)Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: p,c,pc,cp key: cp key_len: 5 ref: NULL rows: 11 filtered: 100.00 Extra: Using index condition1 row in set, 1 warning (0.00 sec)


Cost Estimates

29

5M 50M 500M

Table Scan 53.80 53.80 53.80

(Population) 152.21 34.61 3.81

(Continent) 28.20 28.20 28.20

(Population, Continent)

152.21 34.61 3.81

(Continent, Population)

24.83 16.41 3.81

Where Population >N. Lower is better.


Today’s Agenda

Introduction

World Schema

IMDB Schema

Advanced Topics

1

2

4

1

3


Examples Using IMDB

• Larger than world database schema. • Better to understand performance differences. • License requires data to be regenerated: • http://imdbpy.sourceforge.net/

31

http://imdbpy.sourceforge.net/


mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 2387523 filtered: 0.50 Extra: Using where1 row in set, 1 warning (0.00 sec)

Ignore EXPLAIN time Actual run time is 2.9 seconds (Not shown)


mysql> ALTER TABLE title ADD INDEX (title(100));Query OK, 0 rows affected (4.73 sec)Records: 0 Duplicates: 0 Warnings: 0

mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: refpossible_keys: title key: title key_len: 102 ref: const rows: 4 filtered: 100.00 Extra: Using where1 row in set, 1 warning (0.01 sec)

From 2.9 seconds to 0.0 seconds (!!!)


How does LIKE work?

34

mysql> EXPLAIN SELECT * FROM title WHERE title LIKE 'The Lion %'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: rangepossible_keys: title key: title key_len: 102 ref: NULL rows: 109 filtered: 100.00 Extra: Using where1 row in set, 1 warning (0.01 sec)


LIKE (cont.)

35

mysql> EXPLAIN SELECT * FROM title WHERE title LIKE 'The%'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: rangepossible_keys: title key: title key_len: 102 ref: NULL rows: 325548 filtered: 100.00 Extra: Using where1 row in set, 1 warning (0.00 sec)

VS

Very Close. Estimate is that it is almost as fast to table scan. Run Time is 1.9 sec


Sorting

36

mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King' ORDER BY production_year\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: refpossible_keys: title key: title key_len: 102 ref: const rows: 4 filtered: 100.00 Extra: Using where; Using filesort1 row in set, 1 warning (0.00 sec)


SELECT movie_info.* FROM title INNER JOIN movie_info ON title.id=movie_info.movie_id WHERE title=‘The Lion King’\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: movie_info type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1212187 Extra: NULL********************** 2. row ********************** id: 1 select_type: SIMPLE table: title type: eq_refpossible_keys: PRIMARY,title key: PRIMARY key_len: 4 ref: imdb.movie_info.movie_id rows: 1 Extra: Using where


mysql> ALTER TABLE movie_info ADD INDEX mi (movie_id);Query OK, 0 rows affected (1.95 sec)Records: 0 Duplicates: 0 Warnings: 0


EXPLAIN SELECT movie_info.* FROM title INNER .. WHERE title='The Lion King'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title type: refpossible_keys: PRIMARY,title key: title key_len: 102 ref: const rows: 4 Extra: Using where********************** 2. row ********************** id: 1 select_type: SIMPLE table: movie_info type: refpossible_keys: mi key: mi key_len: 4 ref: imdb.title.id rows: 7 Extra: NULL


mysql> EXPLAIN SELECT * FROM movie_info WHERE movie_id IN (SELECT id FROM title WHERE title = 'The Lion King')\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title type: refpossible_keys: PRIMARY,title key: title key_len: 102 ref: const rows: 4 Extra: Using where********************** 2. row ********************** id: 1 select_type: SIMPLE table: movie_info type: refpossible_keys: mi key: mi key_len: 4 ref: imdb.title.id rows: 7 Extra: NULL


Today’s Agenda

Introduction

World Schema

IMDB Schema

Advanced Topics

1

2

3

1

4


Optimizer Hints

• USE INDEX • FORCE INDEX • IGNORE INDEX • STRAIGHT JOIN

42


Optimizer Hints (cont.)

43

Expanded in MySQL 5.6

mysql> select @@optimizer_switch\G********************** 1. row **********************@@optimizer_switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on1 row in set (0.00 sec)


Statistics Sampling

• Some decisions based on pre-‐computed statistics. • For example, which order to join tables.

• Accuracy of statistics greatly improved in MySQL 5.6. • Feature is called “InnoDB persistent statistics”.

44


Optimizer Decision Visibility

• Intended for MySQL developer usage, and reporting optimizer bugs. • Yet totally practical for advanced usage :)

45

New to MySQL 5.6

SET optimizer_trace="enabled=on";SELECT /* query here */;SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;


Optimizer Trace Output

• Includes why decisions were not made:

46

.. "index": "pc", "ranges": [ "50000000 < Population" ], "index_dives_for_eq_ranges": true, "rowid_ordered": false, "using_mrr": false, "index_only": false, "rows": 24, "cost": 29.81, "chosen": false, "cause": "cost" },


Too many indexes

• Slows down modifications due to maintenance cost. • i.e. Slower INSERT/UPDATE/DELETE

• May also lead to worse decisions being made by optimizer. • Too many similar choices. • Only limited time window to evaluate suitability.

• Space requirements in memory and storage.

47


MySQL 5.7 and beyond

• We are working on a new cost model. • Better adapt to new hardware with different costs (i.e. SSDs have cheap random IO). • Memory buffer aware cost estimates.

48


Cost constants configurable per engine in 5.7

• Previously hard-‐coded constants. • Better adapts to differences in hardware.

49
