Loadays managing my sql with percona toolkit

Managing MySQL with Percona Toolkit

LOADAYS 2012 – April 1st

www.percona.com

Who am I ?

Frédéric Descamps

@lefred

http://about.me/lefred

Percona Consultant

Managing mysql since 3.23 (as far as I remember)

devops believer

www.percona.com

Agenda

● What is Percona Toolkit ?

● What tools are included and what can they do ?

● Quick intro to several tools

● Resources

www.percona.com

Percona Toolkit

● Origin: Maatkit and Aspersa (both projects started by Baron Schwartz with Daniel Nichter's contribution)

● Indispensable tools for MySQL administrators● Opensource GNU GPL v2● https://launchpad.net/percona-toolkit● http://www.percona.com/software/percona-toolkit/● Current version is 2.0.4

www.percona.com

Other Toolkits

www.percona.com

Percona Toolkit ;-)

www.percona.com

More seriously:

www.percona.com

Installation

● Requirements:➢ Perl, DBI, DBD::pgsql, Wine or Mono➢ Better performance with Wine

● Download zip:➢ http://megaupload.com/PTK204➢ It's in Percona's MSN repository

● We are working on a Java version● Easy to run: wine pt-diskstats_win32.exe

www.percona.com

Installation

● Requirements:➢ Perl, DBI, DBD::pgsql, Wine or Mono➢ Better performance with Wine

● Download zip:➢ http://megaupload.com/PTK204➢ It's in Percona's MSN repository

● We are working on a Java version● Easy to run: wine pt-diskstats_win32.exe

www.percona.com

Installation

● Requirements:➢ Perl, DBI, DBD::mysql, Term::ReadKey

● Download tarball, RPM or DEB packages:➢ wget percona.com/get/percona-toolkit.rpm ➢ It's in Percona repository

● Download individual tools:➢ wget percona.com/get/pt-diskstats

11www.percona.com

Know your environment:Hardware & Operating System

12www.percona.com

Know your environment (Hardware & OS)

We need to find the answers to these questions:

● On what kind of machine runs my db ? Physical/Virtual ?● How much memory is used/free ? Does it swap ? Can I add more ?● How many CPUs and CPU cores ? Speed ?● Is this a 64bit machine ?● What kind of RAID controller ? Volumes ? Disks ?● What about network interfaces ? Connections ? Processes ?● What's the Operating System ? Kernel ? Release ?● Which IO scheduler is in use ?● On which filesystem is the data stored ?● Do we have free space ?

13www.percona.com

Summary !

Name: pt-summaryFunction: summarize system information in a nice way

Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-summary.html

14www.percona.com

Taking inventory

$ wget percona.com/get/pt-summary$ chmod +x pt-summary$ ./pt-summary# Percona Toolkit System Summary Report ###################### Date | 2012-02-02 12:25:18 UTC (local TZ: EET +0200) Hostname | db01 Uptime | 42 days, 16:14, 1 user, load average: 1.15, 1.22, 1.08 System | HP; ProLiant DL580 G7; vNot Specified (Rack Mount Chassis) Service Tag | FOSDEM2012 Platform | Linux Release | CentOS release 6.1 (Final) Kernel | 2.6.32-131.21.1.el6.x86_64Architecture | CPU = 64-bit, OS = 64-bit Threading | NPTL 2.12 Compiler | GNU CC version 4.4.5 20110214 (Red Hat 4.4.5-6). SELinux | Disabled Virtualized | No virtualization detected# Processor ################################################## Processors | physical = 4, cores = 40, virtual = 80, hyperthreading = yes Speeds | 80x2397.393 Models | 80xIntel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz

[ Further output omitted ]

15www.percona.com

Know your environment:MySQL

16www.percona.com

Know your environment (MySQL)

We need to find the answers to these questions:

● Version ? Build ?● How many database ? Where is the data directory ?● What about replication ? How many replicas ?● Do you use some Key XtraDB features ?● What are the Key InnoDB settings ?● Which storage engines are used ? ● What types of data fields are used ? Index types ?● Do we use foreign keys ? Views ? Triggers ?● Do we use Full-text indexes ? Partitioning ?● What about Privileges ?

17www.percona.com

Summary... again !

Name: pt-mysql-summaryFunction: Summarize MySQL information in a nice wayUrl: http://www.percona.com/doc/percona-toolkit/2.0/pt-mysql-summary.html

18www.percona.com

Taking inventory (cont.)

$ wget percona.com/get/pt-mysql-summary$ chmod +x pt-mysql-summary$ ./pt-mysql-summary# Percona Toolkit MySQL Summary Report ####################### System time | 2012-02-02 12:24:37 UTC (local TZ: EET +0200)# Instances ################################################## Port Data Directory Socket ===== ========================== ====== /mysql # Report On Port 3306 ######################################## User | root@localhost Time | 2012-02-02 14:24:38 (EET) Hostname | db01 Version | 5.5.18-55-log Percona Built On | Linux x86_64 Started | 2011-12-26 21:09 (up 37+17:15:35) Databases | 6 Datadir | /mysql/ Processes | 10 connected, 4 running Replication | Is a slave, has 1 slaves connected Pidfile | /mysql/mysqld.pid (exists)

[ Further output omitted ]

19www.percona.com

Master ? Slave ?... both ?

Name: pt-slave-findFunction: find and print replication hierarchy tree of MySQL slaves.Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-slave-find.html

20www.percona.com

Inventory of Replicas

$ ./pt-slave-find localhostlocalhostVersion 5.5.18-55-logServer ID 11Uptime 37+17:45:50 (started 2011-12-26T21:09:03)Replication Is a slave, has 1 slaves connected, is not read_onlyFilters Binary logging MIXEDSlave status 0 seconds behind, running, no errorsSlave mode STRICTAuto-increment increment 2, offset 1InnoDB version 1.1.8-rel23.0+- 10.0.0.17 Version 5.5.18-55-log Server ID 12 Uptime 27+21:02:35 (started 2012-01-05T17:52:18) Replication Is a slave, has 1 slaves connected, is not read_only Filters Binary logging MIXED Slave status 0 seconds behind, running, no errors Slave mode STRICT Auto-increment increment 2, offset 2 InnoDB version 1.1.8-rel23.0

21www.percona.com

Where's My Disk I/O Going ?

● Use pt-diskstats as an improved iostat➢ Interactively controllable, slice-and-dice➢ Doesn't lump reads and writes together➢ Meaningful, correct columns

● Use pt-ioprofile to measure & time your I/O

➢ Which files is MySQL using ? How much times does it spend reading, writing, syncing, etc? How many reads/writes? How large are they ?

pt-ioprofile can be dangerous in production

22www.percona.com

Disk I/O

Name: pt-diskstatsFunction: Aggregate and summarize /proc/diskstats

Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-diskstats.html

23www.percona.com

Disk I/O

# ./pt-diskstats

#ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime

{1} sdb 158.0 5.8 0.9 0% 0.8 4.9 392.1 24.3 9.3 51% 0.6 0.7 54% 0 550.1 0.9 0.6 {1} sdb1 158.0 5.8 0.9 0% 0.8 4.9 392.1 24.3 9.3 51% 0.6 0.7 54% 0 550.1 0.9 0.6

{2} sda 0.5 4.0 0.0 0% 0.0 8.0 3.0 59.3 0.2 93% 0.0 0.2 1% 0 3.5 0.0 0.3 {2} sda2 0.0 0.0 0.0 0% 0.0 0.0 1.0 16.0 0.0 75% 0.0 1.0 0% 0 1.0 0.0 1.0 {2} sda5 0.5 4.0 0.0 0% 0.0 8.0 2.0 81.0 0.2 95% 0.0 0.1 1% 0 2.5 0.0 0.2 {2} sdb 150.0 8.7 1.3 3% 1.1 6.9 448.5 40.6 17.8 84% 1.4 0.5 53% 0 598.6 0.6 0.2 {2} sdb1 143.0 8.9 1.2 3% 1.0 6.8 400.5 24.7 9.7 49% 0.7 0.9 51% 0 543.6 1.3 0.5 {2} sdb2 7.0 4.0 0.0 0% 0.1 10.0 48.0 173.0 8.1 98% 0.7 0.3 9% 0 55.0 0.3 0.0

wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt392.1 24.3 9.3 51% 0.6 0.7392.1 24.3 9.3 51% 0.6 0.7

3.0 59.3 0.2 93% 0.0 0.2 1.0 16.0 0.0 75% 0.0 1.0 2.0 81.0 0.2 95% 0.0 0.1448.5 40.6 17.8 84% 1.4 0.5400.5 24.7 9.7 49% 0.7 0.9 48.0 173.0 8.1 98% 0.7 0.3

24www.percona.com

I/O Profile

Name: pt-ioprofileFunction: Watch process IO and print a table of file and I/O activity.

Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-ioprofile.html

25www.percona.com

I/O profile

# ./pt-ioprofileSat Feb 4 13:44:08 CET 2012Tracing process ID 24179 total read write open close lseek filename 0.007547 0.000583 0.004245 0.002181 0.000268 0.000270 /tmp/#sql_5e73_1.MYI 0.000686 0.000000 0.000000 0.000420 0.000266 0.000000 /tmp/#sql_5e73_1.MYD 0.000065 0.000000 0.000024 0.000029 0.000012 0.000000 /var/lib/mysql/fred/db.opt 0.000055 0.000000 0.000000 0.000055 0.000000 0.000000 /var/lib/mysql/mysql/db.opt 0.000046 0.000000 0.000000 0.000046 0.000000 0.000000 /var/lib/mysql/test/db.opt

26www.percona.com

Be intimate with your database

27www.percona.com

Let's try to find the answer to these indiscreet questions:

● How are the indexes used ?● Are there duplicate keys ?● Which queries are eating most of the

resources ?

28www.percona.com

Analyze Schema & Queries

● Duplicate/redundant indexes or foreign keys?➢pt-duplicate-key-checker can tell you that

● Which indexes are unused?➢pt-index-usage can tell you that➢ It can also show which queries have unstable plans, which indexes have alternatives that are never chosen, and lots more

● Bad SQL?➢pt-query-advisor to the rescue

29www.percona.com

Analyze & Profile Your Workload

● Use pt-query-digest to analyze the “slow query log” and show a profile of the workload

● It does the Right Thing by default● Mostly used with slow query logs & tcpdumps

30www.percona.com

The Report

● Each class of query is ranked by execution time● There is a report for each class of query● It's easy to use● It's a great way to start optimizing queries

31www.percona.com

Overview Report# 13903.5s user time, 39.9s system time, 1.78G rss, 1.92G vsz# Current date: Mon Jan 30 20:36:26 2012# Hostname: hp-db01# Files: /var/lib/mysql/db01-slow.log# Overall: 32.98M total, 1.32k unique, 1.73k QPS, 2.06x concurrency ______# Time range: 2012-01-30 11:08:22 to 16:25:40# Attribute total min max avg 95% stddev median# ============ ======= ======= ======= ======= ======= ======= =======# Exec time 39309s 1us 161s 1ms 515us 109ms 30us# Lock time 4471s 0 161s 135us 89us 90ms 0# Rows sent 118.72M 0 1010.29k 3.77 1.96 963.89 0# Rows examine 15.54G 0 3.51M 505.92 1.96 11.61k 0# Rows affecte 1.64M 0 2.85k 0.05 0 1.82 0# Rows read 118.72M 0 1010.29k 3.77 1.96 963.89 0# Bytes sent 52.40G 0 45.18M 1.67k 5.20k 20.30k 537.02# Merge passes 128 0 1 0.00 0 0.00 0# Tmp tables 957.76k 0 4 0.03 0 0.17 0# Tmp disk tbl 297.92k 0 2 0.01 0 0.10 0# Tmp tbl size 5.21T 0 2.22G 169.59k 0 1.78M 0# Query size 8.52G 6 130.40k 277.38 833.10 709.99 174.84# InnoDB:# IO r bytes 672.00k 0 32.00k 0.06 0 35.90 0# IO r ops 42 0 2 0.00 0 0.00 0# IO r wait 19ms 0 2ms 0 0 1us 0# pages distin 329.98M 0 32.08k 28.18 42.48 307.62 4.96# queue wait 0 0 0 0 0 0 0# rec lock wai 3402s 0 161s 277us 0 147ms 0# Boolean:# Filesort 2% yes, 97% no# Filesort on 0% yes, 99% no# Full join 0% yes, 99% no# Full scan 2% yes, 97% no# QC Hit 55% yes, 44% no# Tmp table 2% yes, 97% no# Tmp table on 0% yes, 99% no

32www.percona.com

Profile Report

# Profile# Rank Query ID Response time Calls R/Call Apdx V/M Item# ==== ================== =============== ======== ======= ==== ===== ====# 1 0x844F3DA35A4A426D 8695.6994 22.1% 41470 0.2097 1.00 0.12 SELECT catalog# 2 0xD566217EB8EB1A27 4098.9154 10.4% 48740 0.0841 1.00 0.11 SELECT geoip# 3 0xB0A48B410B3CC151 2684.9671 6.8% 55 48.8176 0.00 34.18 UPDATE SELECT sales_# 4 0x60CDD6CAE6B487BC 2541.5271 6.5% 352 7.2202 0.20 4.01 SELECT catalog catalog_category_product# 5 0xD9DED53EF74C877F 1285.6118 3.3% 5530829 0.0002 1.00 0.00 SELECT op_image# 6 0x1F3D58843008ACEC 1249.1639 3.2% 182 6.8635 0.12 1.65 SELECT catalog_product# 7 0x50175ED6D1B1C8B1 1188.1541 3.0% 153964 0.0077 1.00 0.00 SELECT enterprise# 8 0x70E2BE84868444A4 1035.3475 2.6% 1786 0.5797 0.98 0.16 SELECT sales_order# 9 0x125DFE84FFA50675 891.3327 2.3% 382 2.3333 0.97 11... INSERT UPDATE product# 10 0xBB968DD9BF27E8E5 797.3765 2.0% 153964 0.0052 1.00 0.00 SELECT enterprise

33www.percona.com

Per Query Report# Query 1: 2.25 QPS, 0.47x concurrency, ID 0x844F3DA35A4A426D at byte 4779840203# Attribute pct total min max avg 95% stddev median# ============ === ======= ======= ======= ======= ======= ======= =======# Count 0 41470# Exec time 22 8696s 97ms 3s 210ms 455ms 159ms 155ms# Lock time 0 7s 107us 50ms 177us 247us 272us 159us# Rows sent 1 1.60M 26 50 40.45 49.17 10.74 46.83# Rows examine 16 2.63G 66.43k 66.60k 66.49k 65.68k 0 65.68k# Rows affecte 0 0 0 0 0 0 0 0# Rows read 1 1.60M 26 50 40.45 49.17 10.74 46.83# Bytes sent 0 200.43M 3.39k 6.14k 4.95k 6.01k 1.08k 5.45k# Merge passes 0 0 0 0 0 0 0 0# Tmp tables 4 40.50k 1 1 1 1 0 1# Tmp disk tbl 0 0 0 0 0 0 0 0# Tmp tbl size 12 647.83G 16.00M 16.00M 16.00M 16.00M 0 16.00M# Query size 0 71.78M 1.77k 1.77k 1.77k 1.77k 0 1.77k# InnoDB:# IO r bytes 0 0 0 0 0 0 0 0# IO r ops 0 0 0 0 0 0 0 0# IO r wait 0 0 0 0 0 0 0 0# pages distin 47 158.13M 3.90k 3.92k 3.90k 3.88k 0.00 3.88k# queue wait 0 0 0 0 0 0 0 0# rec lock wai 0 0 0 0 0 0 0 0# Boolean:# Filesort 100% yes, 0% no# Full scan 100% yes, 0% no# Tmp table 100% yes, 0% no# String:# Databases prod# Hosts# InnoDB trxID 3CB60AED (1/0%), 3CB60B3E (1/0%)... 41468 more# Last errno 0# Users prod# Query_time distribution# 1us# 10us# 100us# 1ms# 10ms# 100ms ################################################################# 1s ## 10s+

34www.percona.com

Maintain your environment

35www.percona.com

Log & Analyze Errors ?

● Use pt-deadlock-logger to capture deadlocks

● Use pt-fk-error-logger for foreign key errors

● The tools give you the possibility to log the results to a database table for analysis

36www.percona.com

Monitor dead locks

Name: pt-deadlock-loggerFunction: extract and log MySQL deadlock informationUrl: http://www.percona.com/doc/percona-toolkit/2.0/pt-deadlock-logger.html

● SHOW ENGINE INNODB STATUS reports only the most recent deadlock

● Information in status output lacks details about user, client host, other locks in the deadlock cycle.

● pt-deadlock-logger formats the needed information nicely and saves it

37www.percona.com

Monitor Foreign Key Errors

Name: pt-fk-error-loggerFunction: extract and log MySQL foreign key errors.Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-fk-error-logger.html

● Like pt-deadlock-logger, this tool collects more useful information, formats it nicely and saves it.

38www.percona.com

Non-Blocking ALTER TABLE

● Use pt-online-schema-change to alter tables

➢ It makes a “shadow copy” and swaps them➢Extremely useful for large, long-running ALTER

➢Facebook uses the same technique

39www.percona.com

Validate Upgrades

● Upgrades are a leading cause of downtime➢ Are any queries using different indexes ? Is QEP different ?

➢ Are there new errors or warnings ?➢ Do queries return the same results ?➢ Are there significant response time differences ?

● Validate the results before you upgrade!➢ Capture a snapshot of the data➢ Capture a log of queries➢ Compare the queries on two test servers running the old and the new version

40www.percona.com

Validate upgrades

Name: pt-upgradeFunction: Execute queries on multiple servers and check for differences.Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-upgrade.html

41www.percona.com

pt-upgrade# Query 2: ID 0xE2C8DC0286B7F44E at byte 0 _______________________________# Found 23281 differences in 58207 samples:# checksums 0# column counts 0# column types 0# query times 23281# row counts 0# warning counts 0# warning levels 0# warnings 0# AF002020:3306 AF001885:3306# Errors 0 0# Warnings 0 0# Query_time # sum 34s 70s# min 395us 342us# max 218ms 308ms# avg 587us 1ms# pct_95 725us 3ms# stddev 2ms 4ms# median 541us 690us# row_count # sum 1.75k 1.75k# min 0 0# max 1 1# avg 0.03 0.03# pct_95 0 0# stddev 0.17 0.17# median 0 0

42www.percona.com

Tools for Replication

● Verify replication integrity (pt-table-checksum)● Repair out-of-sync replicas (pt-table-sync)● Measure delay accurately (pt-heartbeat)● Deliberately delay replication (pt-slave-delay)● Watch & restart replication (pt-slave-restart)

43www.percona.com

Verify replication Integrity

Name: pt-table-checksumFunction: perform an online replication consistency check, or checksum MySQL tables efficiently on one or many servers.Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-table-checksum.html

● Use it routinely (mandatory for 95% of MySQL users)● It used to be complicated; no more. Just run it!● Put it in a weekly crontab● Repair differences with pt-table-sync

44www.percona.com

I feel bad...

45www.percona.com

Find problems

● What do you advise me doctor ?● Get the symptoms when it hurts !

46www.percona.com

Get the symptoms when it hurts !

To perform this action, do a check-up when you feel bad:

● pt-stalk● pt-sift

47www.percona.com

Does it hurt ?

Name: pt-stalkFunction: wait for a condition to occur then begin collecting data.Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-stalk.html

48www.percona.com

We need your health parameters

Name: pt-collectFunction: collect information from a server for some period of time.Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html

O B S O L E T E

49www.percona.com

Let me check...

Name: pt-siftFunction: browses files created by pt-stalk.Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html

50www.percona.com

Very useful

Name: pt-mextFunction: Look at many samples of MySQL SHOW GLOBAL STATUS side-by-side

Url: http://www.percona.com/doc/percona-toolkit/pt-mext.html

● Default STATUS shows counter since the MySQL instances started.

● It is more helpful to see a delta of recent activity, to check if a change you applied made a difference.

51www.percona.com

pt-mext

pt-mext -r -- mysqladmin ext -i10 -c5

All values are interesting but you should pay even more attention to:

● Created_tmp_*● Handler_read_*● Innodb_buffer_pool_read*● Innodb_os_log_written● Key_read_requests● Key_reads● Qchache*● Select_full_*join● Select_range● Select_scan● Table_locks_waited● Threads_running

52www.percona.com

Stop Bad Queries

● Use pt-kill to watch for long-running queries and optionally kill them

● Prevent long-running idle transactions too.

53www.percona.com

Percona Toolkit's Future

● Rapid development & improvement of tools● Fully supported via Percona Support contract● Upcoming:

➢pt-*summary redesigned➢pt-stalk “magical fault detection algorithm”➢Redesign of pt-online-schema-change (see

http://www.percona.com/docs/wiki/ptdev:blueprints:redesign-pt-online-schema-change)

➢more goodies•https://blueprints.launchpad.net/percona-toolkit

54www.percona.com

Report bugs & patches

55www.percona.com

Report bugs & patches

Belgian Beers ....... what else !?

Thank you for your participation to LOADAYS

Due to time I was not able to present two new Percona projects related to

High Availability.

If you want to discuss PRM and XtraDB Cluster... I am available

58www.percona.com

PRM

● Percona Replication Manager● Uses Corosync / Pacemaker● Distributed architecture, no SPOF● One master multiple slaves● Slaves maintain a master score attribute

based on replication state● Full stonith/fencing stack● Flexible logic, score based

59www.percona.com

Percona XtraDB Cluster

● Percona Server + Galera● Synchronous replication● Multi-master replication● Multi-threaded slaves● Data consistency across all peers● Automatic nodes provisioning

[email protected]@lefred

We're Hiring! www.percona.com/about-us/careers/