MySQL Monitoring 101

http://ronaldbradford.com

MySQL Monitoring 101

MySQL Monitoring 101Simple stuff to save your bacon

Ronald Bradford

Principal - 42SQL

MySQL Users Conference

Santa Clara - April 2009Version 1.1 22.Apr.2009

http://www.primebase.org




What is Your Situation?How do you know your website is down?





How do you know your website is down?

❖The users will let me know

❖That's somebody else's problem❖Our site is never down❖ Email/SMS/Pager

Is this you?





You have a performance problem now?

❖How long has it been happening?

❖ Is it a new problem or a re-occurring problem?❖Has it gradually become worse over time?

Is this you?



When did you last recover from a backup?

❖We need to think about doing backups

❖We use MySQL replication❖That's somebody else's problem❖We verify our backups worked (not recovery)

Is this you?



Goal





Four Goals to take away

Monitoring a MySQL Server(s)

❖Know what to monitor❖Know how you can monitor

❖ Learn practices to diagnose problems❖Have a foundation of historical information





Overview





Monitoring Overview

❖Hardware

❖ Software

❖MySQL





Hardware





Know Your Physical Resources

❖ CPU

❖ Memory

❖ Disk

❖ Network





Monitoring Hardware Resources

Bare Essentials

❖ vmstat❖ iostat❖ ps❖ netstat❖ ifconfig❖ top






Very Valuable

❖ sar❖ mpstat❖ dstat❖ ethtool






Very Valuable

❖ /proc❖ /cpuinfo❖ /meminfo❖ /loadavg





Identifying Resource Bottlenecks

❖ Which physical resource is being stressed?

❖ Why?





Identifying Resource Bottlenecks

$ vmstat 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s2 s4 in sy cs us sy id 1 0 0 17114496 2962832 63 211 5180 71 71 0 0 0 0 71 0 4763 4625 3183 72 2 26 1 0 0 17115252 2963184 4 6 439 0 0 0 0 0 0 7 0 3251 4021 2795 79 2 20 1 0 0 17115252 2963144 2 1 0 0 0 0 0 0 0 16 0 3748 4427 3049 89 2 10 1 0 0 17115252 2962912 11 11 1360 0 0 0 0 0 0 47 0 4083 4210 2752 79 2 19 0 0 0 17115248 2962744 4 9 1428 0 0 0 0 0 0 10 0 1072 1015 754 17 1 82 1 0 0 17115248 2962664 4 0 0 0 0 0 0 0 0 42 0 3755 3818 2549 69 2 29 1 0 0 17115240 2962520 2 4 246 0 0 0 0 0 0 3 0 3231 3992 2833 79 1 19 1 0 0 17115228 2962400 3 8 1347 0 0 0 0 0 0 10 0 3706 4339 3063 88 2 10 1 0 0 17115220 2962256 16 11 853 0 0 0 0 0 0 52 0 4275 4201 2672 79 2 19 1 0 0 17115220 2962100 3 5 131 0 0 0 0 0 0 7 0 2742 3275 2385 62 2 36 0 0 0 17115220 2962064 3 0 0 0 0 0 0 0 0 29 0 1887 1247 917 19 1 81 1 0 0 17115216 2961664 7 13 1885 0 0 0 0 0 0 18 0 3597 4674 3270 82 2 17 2 0 0 17114344 2961428 75 281 1953 13 13 0 0 0 0 12 0 3697 4746 3203 87 2 11 1 0 0 17114332 2961396 19 14 211 6 6 0 0 0 0 53 0 4409 4476 2862 77 2 21 1 0 0 17115316 2961544 5 7 426 0 0 0 0 0 0 8 0 2658 3478 2371 64 1 34 0 0 0 17115316 2961500 5 0 0 0 0 0 0 0 0 39 0 2310 1802 1268 26 1 73 1 0 0 17115316 2961428 5 7 823 0 0 0 0 0 0 11 0 3454 4473 3096 82 2 16 1 0 0 17115308 2961224 7 11 920 0 0 0 0 0 0 13 0 3569 4377 3024 86 2 12 1 0 0 17115308 2961016 12 9 870 3 3 0 0 0 0 37 0 4687 5227 3250 82 2 17

Can you spot Memory swapping, Disk I/O and CPU usage changes?





Customer Examples

❖ Single CPU Kernel on 4 Processor Server$ uptime 10:32:22 up 196 days, 11:46, 2 users, load average: 3.86, 4.08, 4.26

$ uname -a Linux db1 2.4.20-6 #1 Thu Feb 27 10:06:59 EST 2003 i686 i686 i386 GNU/Linux [root@db1 mysql]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 10 model name : Pentium III (Cascades) stepping : 1 cpu MHz : 701.636 cache size : 2048 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1399.19





Customer Examples

❖ Single CPU Kernel - Disabled from boot loader$ more /boot/grub/grub.conf# grub.conf generated by anaconda## Note that you do not have to rerun grub after making changes to this file# NOTICE: You have a /boot partition. This means that# all kernel and initrd paths are relative to /boot/, eg.# root (hd0,0)# kernel /vmlinuz-version ro root=/dev/sda3# initrd /initrd-version.img#boot=/dev/sdadefault=0timeout=10splashimage=(hd0,0)/grub/splash.xpm.gz root (hd0,0) kernel /vmlinuz-2.4.20-6 ro root=LABEL=/ initrd /initrd-2.4.20-6.img[root@db1 mysql]# ls /boot/vmlinuvmlinux-2.4.20-6 vmlinux-2.4.20-6smp vmlinuz vmlinuz-2.4.20-6 vmlinuz-2.4.20-6smp





Customer Examples

❖ Excessive Network Load (20GB uncompressed backup)$ ethtool eth0 Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: d Wake-on: d Link detected: yes



Customer Examples

❖ Degraded RAID

❖ Who checks RAID status?❖ How is it checked?





Additional Resources

• Identify Bottlenecks• CPU

• Memory

• Check back for additional articles

http://ronaldbradford.com/blog/identifying-resource-bottlenecks-cpu-2009-03-31/http://ronaldbradford.com/blog/identifying-resource-bottlenecks-memory-2009-04-02/



http://ronaldbradford.com/blog/identifying-resource-bottlenecks-memory-2009-04-02/






Software





Know Your Running Software

❖ Operating System

❖ Database

❖ Other





Operating System Specifics

❖Kernel Version❖ 32/64 bit Kernel

❖ 32bit software running on 64bit kernel❖Review all running software❖ Should it be running?





MySQL





MySQL Log Files

❖ Error Log ❖ Could be OS Log

❖ Slow Query Log

1





MySQL Process

❖ Is mysqld running?❖What is the Memory Usage?





MySQL Memory Usage

❖RSS and VSZ

$ ps -eopid,fname,rss,vsz,user,command | grep -e "RSS" -e "mysql" PID COMMAND RSS VSZ USER COMMAND 5463 grep 764 5204 ronald grep -e RSS -e mysql13894 mysqld_s 596 3936 root /bin/sh /usr/bin/mysqld_safe13933 mysqld 4787812 5127208 mysql /usr/sbin/mysqld --basedir=/usr --datadir=/vol/mysql/mysqldata --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock13934 logger 608 3840 root logger -p daemon.err -t mysqld_safe -i -t mysqld

$ ps -eopid,fname,rss,vsz,user,command | grep " mysqld " | grep -v grep | awk '{print $3,$4}'4787820 5127208

http://ronaldbradford.com/blog/are-you-monitoring-rss-vsz-2009-03-08/

2







MySQL System Variables

❖Based on my.cnf configuration❖Customizable in running server

❖ SHOW GLOBAL VARIABLES

http://dev.mysql.com/doc/refman/5.1/en/show-variables.html

3



http://dev.mysql.com/doc/refman/5.1/en/show-status.html




MySQL Status Variables

❖ Provide valuable clues to the state of your servers

❖GLOBAL and SESSION scope

❖ SHOW [GLOBAL|SESSION] STATUS


4







MySQL Connections

❖ SHOW FULL PROCESSLIST


5







MySQL Storage Engines

❖ SHOW ENGINE INNODB STATUS❖ Well incorporated in 5.1 Status Variables


6







Your MySQL Data

❖Know your data❖ Disk footprint❖ Growth

7





Instance Summary

select table_schema,

sum(data_length+index_length)/1024/1024 as total_mb,

sum(data_length)/1024/1024 as data_mb,

sum(index_length)/1024/1024 as index_mb,

count(*) as tables

from information_schema.tables

group by table_schema

order by 2 desc;





Instance Summary

+----------------------------+----------------+----------------+---------------+--------+| table_schema | total_mb | data_mb | index_mb | tables |+----------------------------+----------------+----------------+---------------+--------+| xxxxxxxxxxxxxxxx_531 | 25301.01871777 | 19678.43473339 | 5622.58398438 | 189 || xxxxxxxxxxxxxxxx_528 | 18983.51311207 | 14739.53361988 | 4243.97949219 | 96 || stats | 14803.18925285 | 5469.20292473 | 9333.98632813 | 1082 || xxxxxxxxxxxxxxxx_4 | 9631.98090553 | 7600.90180397 | 2031.07910156 | 1119 || xxxxxxxxxxxxxxxx_501 | 9141.53529739 | 7263.91127396 | 1877.62402344 | 72 || system_logs | 7973.60180283 | 3237.59008408 | 4736.01171875 | 1553 || stats_cpu | 4544.43988991 | 3317.35981178 | 1227.08007813 | 1243 || xxxxxxxxxxxxxxxx_555 | 4024.49787521 | 3436.00080490 | 588.49707031 | 293 || stats_xxxxxxxxx | 3712.71884155 | 2159.92196655 | 1552.79687500 | 1536 |...





Schema Summary

select table_name,engine,row_format, table_rows, avg_row_length,

(data_length+index_length)/1024/1024 as total_mb,

(data_length)/1024/1024 as data_mb,

(index_length)/1024/1024 as index_mb


where table_schema= DATABASE()

order by 6 desc;





Schema Attributes

❖Collations❖Text/Blob

❖ Storage Engines





Schema Attributes

select table_schema,engine,table_collation,

count(*) as tables


where table_schema=DATABASE()

group by table_schema,engine,table_collation;

select table_schema,table_name,column_name,data_type

from information_schema.columns

where table_schema= DATABASE()

and ( data_type LIKE '%TEXT' OR data_type like '%BLOB');





Monitoring





System Monitoring

❖ dstat$ cat dbc_os.sh

#!/bin/sh

DATETIME=`date +%y%m%d.%H%M`

LOG_DIR="/opt/monitor/log/"INTERVAL=5COUNT=`expr 3600 \/ $INTERVAL`LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.txtCSV_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.csvMYSQL_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.mysql.csv

/usr/bin/dstat --time --cpu --mem --disk --net --proc --page --swap --load --nocolor --noheaders --output ${CSV_LOG_FILE} $INTERVAL $COUNT > $LOG_FILE 2>&1sed -e "1,7d" -e "s/-/,/" -e "s/ /,/" ${CSV_LOG_FILE} > ${MYSQL_LOG_FILE}exit 0





System Monitoring

❖ Example dstat output-----time----- ----total-cpu-usage---- ------memory-usage----- -dsk/total-> date/time |usr sys idl wai hiq siq| used buff cach free| read writ>19-04 17:00:01| 7 2 91 1 0 0|1151M 475M 1850M 480M| 51k 859k>19-04 17:00:06| 5 0 95 0 0 0|1168M 475M 1850M 462M|5734B 134k>19-04 17:00:11| 3 0 96 0 0 0|1175M 475M 1850M 456M| 0 168k>19-04 17:00:16| 4 0 92 3 0 0|1184M 475M 1850M 447M| 20k 625k>19-04 17:00:21| 3 0 97 0 0 0|1142M 475M 1850M 488M| 0 307k>19-04 17:00:26| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 9830B>19-04 17:00:31| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 2458B>19-04 17:00:36| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 34k>19-04 17:00:41| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 4915B>19-04 17:00:46| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 31k>19-04 17:00:51| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 42k>19-04 17:00:56| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 30k>19-04 17:01:01| 1 0 99 0 0 0|1151M 475M 1850M 480M| 0 58k>19-04 17:01:06| 0 0 100 0 0 0|1151M 475M 1850M 480M| 0 9011B>19-04 17:01:11| 8 0 91 0 0 0|1153M 475M 1850M 478M| 0 3277B>19-04 17:01:16| 2 0 99 0 0 0|1152M 475M 1850M 479M| 0 37k>19-04 17:01:21| 0 0 100 0 0 0|1152M 475M 1850M 479M| 0 6554B>





MySQL Monitoring

❖ hourly.sh❖ OS❖ vmstat❖ps❖uptime

❖ MySQL❖Variables❖Status❖Process List❖ Innodb Engine Status❖Table Status





MySQL Analysis

❖ statpack - Human Readable STATUS ❖ Database Activity❖ Statement Activity❖ Prepared Statements❖ Admin Commands❖ Thread Cache❖ Table Cache❖ MyISAM Key Cache❖ .... http://www.markleith.co.uk/?p=21



http://www.markleith.co.uk/?p=21

http://www.markleith.co.uk/?p=21



statpack - Read/Write Ratio==================================================================================================== Uptime: 17 days 17 hours 22 mins Snapshot Period 1: 59 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Statement Activity ====================================================================================================

SELECT: 13,503,876 3,798.56 4,298,170,239 (94.25%) INSERT: 91,101 25.63 25,327,062 (0.56%) UPDATE: 782,004 219.97 220,640,296 (4.84%) DELETE: 9,674 2.72 2,485,643 (0.05%) REPLACE: 0 0.00 4,980 (0.00%) INSERT ... SELECT: 0 0.00 0 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 46,422 13.06 13,700,478 (0.30%) ROLLBACK: 0 0.00 13 (0.00%)



statpack - Unnecessary SQL

==================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Statement Activity ====================================================================================================

SELECT: 16,042 267.37 8,177,050 (46.03%) INSERT: 5,838 97.30 1,826,616 (10.28%) UPDATE: 1,109 18.48 738,546 (4.16%) DELETE: 2,018 33.63 1,374,983 (7.74%) REPLACE: 0 0.00 0 (0.00%) INSERT ... SELECT: 0 0.00 27 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 5,708 95.13 2,161,232 (12.17%) ROLLBACK: 5,746 95.77 3,485,828 (19.62%)





statpack - Ineffective Query Cache

==================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Query Cache ====================================================================================================

QCache Hits / SELECT: 11.11% QCache Hit/Qcache Insert: 31.24% Qcache Hits/Invalidations: 0.00% SELECTs: 16,042 267.37 8,177,050 Query Cache Hits: 1,257 20.95 1,022,301 Query Cache Inserts: 4,181 69.68 2,250,062 Queries Not Cached: 11,864 197.73 5,932,162 Cache Low Memory Prunes: 0 0.00 0 Total Cache Blocks: -421 -7.02 12,051 Queries In Cache: -595 -9.92 4,194 Cache Free Blocks: 770 12.83 3,628



Graphing Options

❖RRDtool❖ gnuplot

❖Google Charts API

❖ vmplot.sh❖ http://www.bigdbahead.com/?p=302❖ http://ronaldbradford.com/blog/extending-

vmplot-2009-03-31/



http://www.bigdbahead.com/?p=302

http://www.bigdbahead.com/?p=302

http://ronaldbradford.com/blog/extending-vmplot-2009-03-31/






vmplot.sh output





Always graph your results

40%

30%

50%

60%

70%

43%





Products





MySQL Enterprise Monitor

❖Commercial Product - $$$❖Developed by Sun/MySQL

http://www.mysql.com/products/enterprise/monitor.html







Nagios

❖Network and Application Monitoring❖Open Source

http://www.nagios.org/







Cacti

❖Generic Network Graphing❖Uses RRDtool

❖MySQL specific templates

http://www.cacti.net/http://code.google.com/p/mysql-cacti-templates/





http://code.google.com/p/mysql-cacti-templates/

http://code.google.com/p/mysql-cacti-templates/



Additional

❖Hyperic❖Ganglia

❖Munin❖Big Brother❖MONyog❖ EM7❖ Zenoss





External Services

❖ Pingdom❖WebPerform





What's missing

❖ Some Standards❖Active Community

❖ Integration for monitoring all stack products

❖Application specific metrics❖ e.g. Number of new customers per hour❖ e.g. Total Sales for day





Professional Help is Available

❖Two decades IT expertise❖ 10 years in MySQL❖ System/Data Architecture❖ Database Performance and Tuning❖ High Availability and Scalability❖ Education and Training






MySQL Monitoring 101

Technology

mysql monitoring

db1 mysql

mysql servers

mysql replication thats

hardware http

goal http

root hd0

kernel vmlinuzversion