MySQL Monitoring 101
Post on 11-May-2015
20021 Views
Preview:
DESCRIPTION
Transcript
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Monitoring 101Simple stuff to save your bacon
Ronald Bradford
Principal - 42SQL
MySQL Users Conference
Santa Clara - April 2009Version 1.1 22.Apr.2009
http://ronaldbradford.com
MySQL Monitoring 101
What is Your Situation?How do you know your website is down?
http://ronaldbradford.com
MySQL Monitoring 101
How do you know your website is down?
❖The users will let me know
❖That's somebody else's problem❖Our site is never down❖ Email/SMS/Pager
Is this you?
http://ronaldbradford.com
MySQL Monitoring 101
You have a performance problem now?
❖How long has it been happening?
❖ Is it a new problem or a re-occurring problem?❖Has it gradually become worse over time?
Is this you?
http://ronaldbradford.com
MySQL Monitoring 101
When did you last recover from a backup?
❖We need to think about doing backups
❖We use MySQL replication❖That's somebody else's problem❖We verify our backups worked (not recovery)
Is this you?
http://ronaldbradford.com
MySQL Monitoring 101
Four Goals to take away
Monitoring a MySQL Server(s)
❖Know what to monitor❖Know how you can monitor
❖ Learn practices to diagnose problems❖Have a foundation of historical information
http://ronaldbradford.com
MySQL Monitoring 101
Overview
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Overview
❖Hardware
❖ Software
❖MySQL
http://ronaldbradford.com
MySQL Monitoring 101
Hardware
http://ronaldbradford.com
MySQL Monitoring 101
Know Your Physical Resources
❖ CPU
❖ Memory
❖ Disk
❖ Network
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Hardware Resources
Bare Essentials
❖ vmstat❖ iostat❖ ps❖ netstat❖ ifconfig❖ top
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Hardware Resources
Very Valuable
❖ sar❖ mpstat❖ dstat❖ ethtool
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring Hardware Resources
Very Valuable
❖ /proc❖ /cpuinfo❖ /meminfo❖ /loadavg
http://ronaldbradford.com
MySQL Monitoring 101
Identifying Resource Bottlenecks
❖ Which physical resource is being stressed?
❖ Why?
http://ronaldbradford.com
MySQL Monitoring 101
Identifying Resource Bottlenecks
$ vmstat 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s2 s4 in sy cs us sy id 1 0 0 17114496 2962832 63 211 5180 71 71 0 0 0 0 71 0 4763 4625 3183 72 2 26 1 0 0 17115252 2963184 4 6 439 0 0 0 0 0 0 7 0 3251 4021 2795 79 2 20 1 0 0 17115252 2963144 2 1 0 0 0 0 0 0 0 16 0 3748 4427 3049 89 2 10 1 0 0 17115252 2962912 11 11 1360 0 0 0 0 0 0 47 0 4083 4210 2752 79 2 19 0 0 0 17115248 2962744 4 9 1428 0 0 0 0 0 0 10 0 1072 1015 754 17 1 82 1 0 0 17115248 2962664 4 0 0 0 0 0 0 0 0 42 0 3755 3818 2549 69 2 29 1 0 0 17115240 2962520 2 4 246 0 0 0 0 0 0 3 0 3231 3992 2833 79 1 19 1 0 0 17115228 2962400 3 8 1347 0 0 0 0 0 0 10 0 3706 4339 3063 88 2 10 1 0 0 17115220 2962256 16 11 853 0 0 0 0 0 0 52 0 4275 4201 2672 79 2 19 1 0 0 17115220 2962100 3 5 131 0 0 0 0 0 0 7 0 2742 3275 2385 62 2 36 0 0 0 17115220 2962064 3 0 0 0 0 0 0 0 0 29 0 1887 1247 917 19 1 81 1 0 0 17115216 2961664 7 13 1885 0 0 0 0 0 0 18 0 3597 4674 3270 82 2 17 2 0 0 17114344 2961428 75 281 1953 13 13 0 0 0 0 12 0 3697 4746 3203 87 2 11 1 0 0 17114332 2961396 19 14 211 6 6 0 0 0 0 53 0 4409 4476 2862 77 2 21 1 0 0 17115316 2961544 5 7 426 0 0 0 0 0 0 8 0 2658 3478 2371 64 1 34 0 0 0 17115316 2961500 5 0 0 0 0 0 0 0 0 39 0 2310 1802 1268 26 1 73 1 0 0 17115316 2961428 5 7 823 0 0 0 0 0 0 11 0 3454 4473 3096 82 2 16 1 0 0 17115308 2961224 7 11 920 0 0 0 0 0 0 13 0 3569 4377 3024 86 2 12 1 0 0 17115308 2961016 12 9 870 3 3 0 0 0 0 37 0 4687 5227 3250 82 2 17
Can you spot Memory swapping, Disk I/O and CPU usage changes?
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Single CPU Kernel on 4 Processor Server$ uptime 10:32:22 up 196 days, 11:46, 2 users, load average: 3.86, 4.08, 4.26
$ uname -a Linux db1 2.4.20-6 #1 Thu Feb 27 10:06:59 EST 2003 i686 i686 i386 GNU/Linux [root@db1 mysql]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 10 model name : Pentium III (Cascades) stepping : 1 cpu MHz : 701.636 cache size : 2048 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1399.19
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Single CPU Kernel - Disabled from boot loader$ more /boot/grub/grub.conf# grub.conf generated by anaconda## Note that you do not have to rerun grub after making changes to this file# NOTICE: You have a /boot partition. This means that# all kernel and initrd paths are relative to /boot/, eg.# root (hd0,0)# kernel /vmlinuz-version ro root=/dev/sda3# initrd /initrd-version.img#boot=/dev/sdadefault=0timeout=10splashimage=(hd0,0)/grub/splash.xpm.gz root (hd0,0) kernel /vmlinuz-2.4.20-6 ro root=LABEL=/ initrd /initrd-2.4.20-6.img[root@db1 mysql]# ls /boot/vmlinuvmlinux-2.4.20-6 vmlinux-2.4.20-6smp vmlinuz vmlinuz-2.4.20-6 vmlinuz-2.4.20-6smp
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Excessive Network Load (20GB uncompressed backup)$ ethtool eth0 Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: d Wake-on: d Link detected: yes
http://ronaldbradford.com
MySQL Monitoring 101
Customer Examples
❖ Degraded RAID
❖ Who checks RAID status?❖ How is it checked?
http://ronaldbradford.com
MySQL Monitoring 101
Additional Resources
• Identify Bottlenecks• CPU
• Memory
• Check back for additional articles
http://ronaldbradford.com/blog/identifying-resource-bottlenecks-cpu-2009-03-31/http://ronaldbradford.com/blog/identifying-resource-bottlenecks-memory-2009-04-02/
http://ronaldbradford.com
MySQL Monitoring 101
Software
http://ronaldbradford.com
MySQL Monitoring 101
Know Your Running Software
❖ Operating System
❖ Database
❖ Other
http://ronaldbradford.com
MySQL Monitoring 101
Operating System Specifics
❖Kernel Version❖ 32/64 bit Kernel
❖ 32bit software running on 64bit kernel❖Review all running software❖ Should it be running?
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Log Files
❖ Error Log ❖ Could be OS Log
❖ Slow Query Log
1
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Process
❖ Is mysqld running?❖What is the Memory Usage?
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Memory Usage
❖RSS and VSZ
$ ps -eopid,fname,rss,vsz,user,command | grep -e "RSS" -e "mysql" PID COMMAND RSS VSZ USER COMMAND 5463 grep 764 5204 ronald grep -e RSS -e mysql13894 mysqld_s 596 3936 root /bin/sh /usr/bin/mysqld_safe13933 mysqld 4787812 5127208 mysql /usr/sbin/mysqld --basedir=/usr --datadir=/vol/mysql/mysqldata --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock13934 logger 608 3840 root logger -p daemon.err -t mysqld_safe -i -t mysqld
$ ps -eopid,fname,rss,vsz,user,command | grep " mysqld " | grep -v grep | awk '{print $3,$4}'4787820 5127208
http://ronaldbradford.com/blog/are-you-monitoring-rss-vsz-2009-03-08/
2
http://ronaldbradford.com
MySQL Monitoring 101
MySQL System Variables
❖Based on my.cnf configuration❖Customizable in running server
❖ SHOW GLOBAL VARIABLES
http://dev.mysql.com/doc/refman/5.1/en/show-variables.html
3
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Status Variables
❖ Provide valuable clues to the state of your servers
❖GLOBAL and SESSION scope
❖ SHOW [GLOBAL|SESSION] STATUS
http://dev.mysql.com/doc/refman/5.1/en/show-status.html
4
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Connections
❖ SHOW FULL PROCESSLIST
http://dev.mysql.com/doc/refman/5.1/en/show-status.html
5
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Storage Engines
❖ SHOW ENGINE INNODB STATUS❖ Well incorporated in 5.1 Status Variables
http://dev.mysql.com/doc/refman/5.1/en/show-status.html
6
http://ronaldbradford.com
MySQL Monitoring 101
Your MySQL Data
❖Know your data❖ Disk footprint❖ Growth
7
http://ronaldbradford.com
MySQL Monitoring 101
Instance Summary
select table_schema,
sum(data_length+index_length)/1024/1024 as total_mb,
sum(data_length)/1024/1024 as data_mb,
sum(index_length)/1024/1024 as index_mb,
count(*) as tables
from information_schema.tables
group by table_schema
order by 2 desc;
http://ronaldbradford.com
MySQL Monitoring 101
Instance Summary
+----------------------------+----------------+----------------+---------------+--------+| table_schema | total_mb | data_mb | index_mb | tables |+----------------------------+----------------+----------------+---------------+--------+| xxxxxxxxxxxxxxxx_531 | 25301.01871777 | 19678.43473339 | 5622.58398438 | 189 || xxxxxxxxxxxxxxxx_528 | 18983.51311207 | 14739.53361988 | 4243.97949219 | 96 || stats | 14803.18925285 | 5469.20292473 | 9333.98632813 | 1082 || xxxxxxxxxxxxxxxx_4 | 9631.98090553 | 7600.90180397 | 2031.07910156 | 1119 || xxxxxxxxxxxxxxxx_501 | 9141.53529739 | 7263.91127396 | 1877.62402344 | 72 || system_logs | 7973.60180283 | 3237.59008408 | 4736.01171875 | 1553 || stats_cpu | 4544.43988991 | 3317.35981178 | 1227.08007813 | 1243 || xxxxxxxxxxxxxxxx_555 | 4024.49787521 | 3436.00080490 | 588.49707031 | 293 || stats_xxxxxxxxx | 3712.71884155 | 2159.92196655 | 1552.79687500 | 1536 |...
http://ronaldbradford.com
MySQL Monitoring 101
Schema Summary
select table_name,engine,row_format, table_rows, avg_row_length,
(data_length+index_length)/1024/1024 as total_mb,
(data_length)/1024/1024 as data_mb,
(index_length)/1024/1024 as index_mb
from information_schema.tables
where table_schema= DATABASE()
order by 6 desc;
http://ronaldbradford.com
MySQL Monitoring 101
Schema Attributes
❖Collations❖Text/Blob
❖ Storage Engines
http://ronaldbradford.com
MySQL Monitoring 101
Schema Attributes
select table_schema,engine,table_collation,
count(*) as tables
from information_schema.tables
where table_schema=DATABASE()
group by table_schema,engine,table_collation;
select table_schema,table_name,column_name,data_type
from information_schema.columns
where table_schema= DATABASE()
and ( data_type LIKE '%TEXT' OR data_type like '%BLOB');
http://ronaldbradford.com
MySQL Monitoring 101
Monitoring
http://ronaldbradford.com
MySQL Monitoring 101
System Monitoring
❖ dstat$ cat dbc_os.sh
#!/bin/sh
DATETIME=`date +%y%m%d.%H%M`
LOG_DIR="/opt/monitor/log/"INTERVAL=5COUNT=`expr 3600 \/ $INTERVAL`LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.txtCSV_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.csvMYSQL_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.mysql.csv
/usr/bin/dstat --time --cpu --mem --disk --net --proc --page --swap --load --nocolor --noheaders --output ${CSV_LOG_FILE} $INTERVAL $COUNT > $LOG_FILE 2>&1sed -e "1,7d" -e "s/-/,/" -e "s/ /,/" ${CSV_LOG_FILE} > ${MYSQL_LOG_FILE}exit 0
http://ronaldbradford.com
MySQL Monitoring 101
System Monitoring
❖ Example dstat output-----time----- ----total-cpu-usage---- ------memory-usage----- -dsk/total-> date/time |usr sys idl wai hiq siq| used buff cach free| read writ>19-04 17:00:01| 7 2 91 1 0 0|1151M 475M 1850M 480M| 51k 859k>19-04 17:00:06| 5 0 95 0 0 0|1168M 475M 1850M 462M|5734B 134k>19-04 17:00:11| 3 0 96 0 0 0|1175M 475M 1850M 456M| 0 168k>19-04 17:00:16| 4 0 92 3 0 0|1184M 475M 1850M 447M| 20k 625k>19-04 17:00:21| 3 0 97 0 0 0|1142M 475M 1850M 488M| 0 307k>19-04 17:00:26| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 9830B>19-04 17:00:31| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 2458B>19-04 17:00:36| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 34k>19-04 17:00:41| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 4915B>19-04 17:00:46| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 31k>19-04 17:00:51| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 42k>19-04 17:00:56| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 30k>19-04 17:01:01| 1 0 99 0 0 0|1151M 475M 1850M 480M| 0 58k>19-04 17:01:06| 0 0 100 0 0 0|1151M 475M 1850M 480M| 0 9011B>19-04 17:01:11| 8 0 91 0 0 0|1153M 475M 1850M 478M| 0 3277B>19-04 17:01:16| 2 0 99 0 0 0|1152M 475M 1850M 479M| 0 37k>19-04 17:01:21| 0 0 100 0 0 0|1152M 475M 1850M 479M| 0 6554B>
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Monitoring
❖ hourly.sh❖ OS❖ vmstat❖ps❖uptime
❖ MySQL❖Variables❖Status❖Process List❖ Innodb Engine Status❖Table Status
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Analysis
❖ statpack - Human Readable STATUS ❖ Database Activity❖ Statement Activity❖ Prepared Statements❖ Admin Commands❖ Thread Cache❖ Table Cache❖ MyISAM Key Cache❖ .... http://www.markleith.co.uk/?p=21
http://ronaldbradford.com
MySQL Monitoring 101
statpack - Read/Write Ratio==================================================================================================== Uptime: 17 days 17 hours 22 mins Snapshot Period 1: 59 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Statement Activity ====================================================================================================
SELECT: 13,503,876 3,798.56 4,298,170,239 (94.25%) INSERT: 91,101 25.63 25,327,062 (0.56%) UPDATE: 782,004 219.97 220,640,296 (4.84%) DELETE: 9,674 2.72 2,485,643 (0.05%) REPLACE: 0 0.00 4,980 (0.00%) INSERT ... SELECT: 0 0.00 0 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 46,422 13.06 13,700,478 (0.30%) ROLLBACK: 0 0.00 13 (0.00%)
http://ronaldbradford.com
MySQL Monitoring 101
statpack - Unnecessary SQL
==================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Statement Activity ====================================================================================================
SELECT: 16,042 267.37 8,177,050 (46.03%) INSERT: 5,838 97.30 1,826,616 (10.28%) UPDATE: 1,109 18.48 738,546 (4.16%) DELETE: 2,018 33.63 1,374,983 (7.74%) REPLACE: 0 0.00 0 (0.00%) INSERT ... SELECT: 0 0.00 27 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 5,708 95.13 2,161,232 (12.17%) ROLLBACK: 5,746 95.77 3,485,828 (19.62%)
http://ronaldbradford.com
MySQL Monitoring 101
statpack - Ineffective Query Cache
==================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ==================================================================================================== Variable Delta/Percentage Per Second Total==================================================================================================== Query Cache ====================================================================================================
QCache Hits / SELECT: 11.11% QCache Hit/Qcache Insert: 31.24% Qcache Hits/Invalidations: 0.00% SELECTs: 16,042 267.37 8,177,050 Query Cache Hits: 1,257 20.95 1,022,301 Query Cache Inserts: 4,181 69.68 2,250,062 Queries Not Cached: 11,864 197.73 5,932,162 Cache Low Memory Prunes: 0 0.00 0 Total Cache Blocks: -421 -7.02 12,051 Queries In Cache: -595 -9.92 4,194 Cache Free Blocks: 770 12.83 3,628
http://ronaldbradford.com
MySQL Monitoring 101
Graphing Options
❖RRDtool❖ gnuplot
❖Google Charts API
❖ vmplot.sh❖ http://www.bigdbahead.com/?p=302❖ http://ronaldbradford.com/blog/extending-
vmplot-2009-03-31/
http://ronaldbradford.com
MySQL Monitoring 101
vmplot.sh output
http://ronaldbradford.com
MySQL Monitoring 101
Always graph your results
40%
30%
50%
60%
70%
43%
http://ronaldbradford.com
MySQL Monitoring 101
Products
http://ronaldbradford.com
MySQL Monitoring 101
MySQL Enterprise Monitor
❖Commercial Product - $$$❖Developed by Sun/MySQL
http://www.mysql.com/products/enterprise/monitor.html
http://ronaldbradford.com
MySQL Monitoring 101
Nagios
❖Network and Application Monitoring❖Open Source
http://www.nagios.org/
http://ronaldbradford.com
MySQL Monitoring 101
Cacti
❖Generic Network Graphing❖Uses RRDtool
❖MySQL specific templates
http://www.cacti.net/http://code.google.com/p/mysql-cacti-templates/
http://ronaldbradford.com
MySQL Monitoring 101
Additional
❖Hyperic❖Ganglia
❖Munin❖Big Brother❖MONyog❖ EM7❖ Zenoss
http://ronaldbradford.com
MySQL Monitoring 101
External Services
❖ Pingdom❖WebPerform
http://ronaldbradford.com
MySQL Monitoring 101
What's missing
❖ Some Standards❖Active Community
❖ Integration for monitoring all stack products
❖Application specific metrics❖ e.g. Number of new customers per hour❖ e.g. Total Sales for day
http://ronaldbradford.com
MySQL Monitoring 101
Professional Help is Available
❖Two decades IT expertise❖ 10 years in MySQL❖ System/Data Architecture❖ Database Performance and Tuning❖ High Availability and Scalability❖ Education and Training
http://ronaldbradford.com
top related