Mountain Moodle Moot -- MySQL Server & Query Tuing

Query Tuning, System Tuning, and Little Known Tricks Using MySQL

Workbench [email protected]

@stokerSlideshare.net/davestokes

Mountain Moodle Moot

Databases Can Be a Mystery, Misery, or Magnificent

● You do not need to be a DBA● Nothing wrong with being a DBA

● You should think in sets● 'Relational Calculus for $500, Alex.'

● Let the Database do the Heavy Lifting

● Select What you need for speed

● Yes, YOU do need to do backups● And know how to restore from them

Server Tuning

Run latest release of MySQL Better performance Bug Fixes New features

Don't skimp on hardware Memory is better ROI than disk New SSDs are amazing FusionIO cards OMG

Configuration

Configuration should match hardware Remove Query Cache Nasty Mutex But what if you have 'repeatable data', i.e. sports score

Memcached In Application

Know your parameters Session, Global Needs reboot to change or on the fly

I_S, P_S, SYS

Information_Schema Performance_Schema Similar to V$ variables for Oracle 2-5% performance overhead

SYS Predefined for most common questions

You will not need them (now) but know that they are there when you need to eek out last %

Intro to Query Tuning

Reading from disk to memory is 100,000 slower than memory alone

'SELECT *' on a wide table with many col-umns when you need only two columns MUL-TIPLED by N thousands access a minute adds up

Read what you need for speed, no SELECT * Use joins to get desired sets

Our First Query

SELECT City.Name, Country.name, City.PopulationFROM CityJOIN Country ON

(Country.code = City.CountryCode)

Results

Our First Query

SELECT City.Name, Country.name, City.PopulationFROM CityJOIN Country ON

(Country.code = City.CountryCode)

Columns Selected (note Name & Name)

Our First Query

SELECT City.Name, Country.name, City.PopulationFROM CityJOIN Country ON

(Country.code = City.CountryCode)

Tables used

Venn Diagram

City is A, Country is B

Make query match diagram

SELECT a.Name, b.name, a.PopulationFROM City AS a JOIN Country AS b

ON (b.code = a.CountryCode)

Make query match diagram

SELECT a.Name, b.name, a.PopulationFROM City a JOIN Country b

ON (b.code = a.CountryCode)

Can also use AS to alias column names

Aliased Column Names

SELECT a.Name as 'City', b.name as 'Country' , a.Population as 'Pop.'FROM City aJOIN Country b ON (b.code = a.CountryCode)

Updated Results

Query Plan

EXPLAIN

EXPLAIN is pre-pended to a SQL statement and produces output like:

EXPLAIN

We see how many roads to be read:

Guess on rows needing to be read

4079 x 1 = 4079 rows to be read

EXPLAIN

In English – We read all the rows in the City table and find via an index to find the match in the Country table:

So what is an index?

http://en.wikipedia.org/wiki/Database_indexA database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and the use of more storage space to main-tain the extra copy of data. Indexes are used to quickly locate data without having to search every row in a database table every time a database table is accessed. Indexes can be cre-ated using one or more columns of a database table, provid-ing the basis for both rapid random lookups and efficient ac-cess of ordered records.

Uh, in English

Indexes let you go directly to the record you want Rolodex Card Catalog

Overhead Maintenance Sometimes you

DO have read all the table

B-Tree Index

A Peek at an Index

SHOW INDEX FROM <table>;

Statistics

Optimizer uses table statistics to generate cost data.

Out of data stats can make optimizer plan poorly

See 14.13.17.1 Con-figuring Persistent Optimizer Statistics Parameters in MySQL Manual

Run ANALYSE TA-BLE to update stats

Run ANALYSE TA-BLE after adding a new index

Store stat data at SHUTDOWN, Re-store at REBOOT

Lets look at the tables

EER Map of tables in World Database

Just City and Country

The two tables we have been using

Just City and Country

JOIN Country b ON (b.code = a.CountryCode)

Without Indexes

Plural of Moose List of definitions not alphabetized Duplicates

Compound Index

CREATE TABLE test ( id INT NOT NULL, last_name CHAR(30) NOT NULL, first_name CHAR(30) NOT NULL, PRIMARY KEY (id), INDEX name (last_name,first_name));

This index can use be used for searching on last_name + first_name or just last_name

Multi Column Index

CREATE INDEX CityStateZip ONCity (City, State, Zip);

This index can be used to search onCity + State + Zip, City + State, or City

This index can not be used to search onZip, State + Zip

Speeds up finding Zip when used in City + State

Why is it Important to Optimize?

MySQL, unlike other databases, has no way to lock a query plan. That means each time a query is submitted it goes through the opti-mizer.

Optimize Data

Ideally you want to allocate just enough stor-age for any possible record.

Have to use best judgment Human age range 0-999 Postal codes Address lines Phone numbers 01-123-123-1234 and extension ?! PROCEDURE ANALYSE will examine records, make suggestion

Not all Integers BIGINTs Not all text fields VARCHAR(255) Excess column widths are extra baggage

Integers

Type Storage (bytes)

Minimum Signed

Maximum Signed

Minimum Unsigned

Maximum Unsigned

TINYINT 1 -128 127 0 255

SMALLINT 2 -32768 32768 0 65535

MEDIUMINT

3 -8388608 8388607 0 16777215

INT 4 -2147483648 2147483647 0 4294967295

BIGINT 8 -9223372036854

775808

9223372036854775807

0 18446744073709551615

Are you really going to have 18,446,744,073,709,551,615 student id numbers?

Making it up in volume

Consider the worst case Data needs to be read off disk Transferred into memory Sent across network to application server Application server loads data into memory Application sends that data over network to end user

INT VS BIGINT – 4 extra bytes each times X number of records X number of users

Flexibility

Making changes to a schema is costly. It is a lot better for 5.6 Much better for 5.7

Must update replication masters and slaves (double check if done through replication)

Gender Example 2, 3, 17, or 58+

Replication

Changes made to master Replicated to slave server(s) Three threads

BINLOG dump thread Slave I/O thread Slave SQL thread

Read/Write Splits

Writes to Master Reads from slave(s)

Types of Replication

Statement Based – SQL Row Based – Binary changes after SQL exe-

cuted Mixed – Uses Statement Based but switches

when advantageous

Smarter Row Based Replication

MySQL 5.6 and later No need to copy unmodified parts of a row

Global Transaction ID

Previously had to match Master's log position to when starting replication

Each transaction unique Point slave at master and ask it to catch up

MySQL Utilities

Free, open sources Python scripts to auto-mate functions

mysqldbcompare Compare databases on two servers or the same server Compare definitions and data Generate a difference report Generate SQL transformation statements

mysqldbcopy Copy databases between servers Clone databases on the same server Supports rename

mysqldbexport Export metadata and/or data from one or more databases Formats: SQL, CSV, TAB, Grid, Vertical

mysqldbimport Import metadata and data from one or more files Reads all formats from mysqldbexport

mysqldiff Compare object definitions Generate a difference report

These utilities are those designed to perform general operations such as

reporting and searching.

mysqldiskusage Show disk usage for databases Generate reports in SQL, CSV, TAB, Grid, Vertical

mysqlfrm Reads .frm files, optionally in byte-by-byte diagnostic mode Generates CREATE statements from table definition data

mysqlindexcheck Read indexes for one or more tables Check for redundant and duplicate indexes Generate reports in SQL, CSV, TAB, Grid, Vertical

mysqlmetagrep Search metadata Regexp, database search Generate SQL statement for search query

mysqlprocgrep Search process information Generate SQL statement for search Kill processes that match query

mysqluserclone Clone a user account, to the same or different server Show user grants

mysqluc Command line client for running MySQL Utilities Allows a persistent connection to a MySQL Server Tab completion for utility names and options Allows calling the commands with shorter names, such as using "serverinfo" instead of mysqlserverinfo

These utilities are those designed to support replication and high availability operations

for MySQL servers.

mysqlfailover Provides automatic failover on a replication topology Uses Global Transaction Identifiers (GTID, MySQL Server 5.6.5+)

mysqlreplicate Setup replication Start from beginning, current, specific binlog, pos

mysqlrplms Provides round-robin multi-source replication (a slave server continually cycles through multiple masters in order to store

a consolidated data set) Uses Global Transaction Identifiers (GTID, MySQL Server 5.6.9+)

mysqlrpladmin Administers the replication topology Allows recovery of the master Commands include elect, failover, gtid, health, start, stop, and switchover

mysqlrplcheck Check replication configuration Tests binary logging on master

mysqlrplshow Show slaves attached to master Can search recursively Show the replication topology as a graph or list

mysqlrplsync Check data consistency between servers in a replicated setup Uses Global Transaction Identifiers (GTID) Requires MySQL Server 5.6.14 and higher

MySQL Fabric

Also written in Python and free Sharding High

Availability

Exercise

Mysqldbcopy –source=root:pass@localhost --destination=root:pass@localhost world:xworld

Use MySQL Utility mysqldbcopy to make a copy of world database

Exercise

Using world database, run query from earlier to see query plan.

SELECT a.Name as 'City', b.name as 'Country' , a.Population as 'Pop.'FROM City aJOIN Country b ON (b.code = a.CountryCode);

Remove Indexes

Use worldx schema which redefines the World database without indexes

Exercise

Using worldx database again, run query from earlier to see query plan to see cost.

SELECT a.Name as 'City', b.name as 'Country' , a.Population as 'Pop.'FROM City aJOIN Country b ON (b.code = a.CountryCode)

Back to World

How does a 'qualifier' effect query plan?

SELECT a.Name as 'City', b.name as 'Country' , a.Population as 'Pop.'FROM City aJOIN Country b ON (b.code = a.CountryCode)WHERE a.Population > 500000;

Back to World Again

How does a second 'qualifier' effect query plan?

SELECT a.Name as 'City', b.name as 'Country' , a.Population as 'Pop.'FROM City aJOIN Country b ON (b.code = a.CountryCode)WHERE a.Population > 500000ORDER By b.name;

Run those last two queries on worldx

Did indexes help/hurt? Are the qualifiers run before/after join?

Moodle 2.7

A quick look at the schema

More Info on Tuning

High Performance MySQL - 3rd Edition Schwartz, Zaitsev, & Tkachenko

Effective MySql Optimizing Sql Statements Bradford

MySQL Marinate

Free on-line Virtual class Boston MySQL Users Group One Chapter a week Homework Checked by

Mozilla's senior DBA

MySQL Training from Oracle

Introduction to MySQL MySQL Quickstart Fundamentals MySQL for Beginners MySQL for Database Administrators MySQL for Developers MySQL Performance Tuning MySQL High Availability MySQL Cluster MySQL Developer Techniques MySQL and PHP - Developing Dynamic Web Applica-

tions MySQL Advanced Stored Procedures

MySQL Central @ Oracle Open World

Keynotes from Booking.com, Twitter & Dropbox 50+ session, tutorials, BOFs and reception

Q/A

[email protected] @stoker

MySQL Central at Oracle Open WorldSan Francisco Septermber 29th to October 3rd