The 5 Minute DBA-DBA Skills for Non-DBA

THE 5 MINUTE DBA DBA Skills for Non-DBA’s

Matt Yonkovit ,VP (Consulting) – PerconaPercona University, Washington DC

September 12, 2013

DBA’s are hard to find!

Company Confidential December 2010

-2-

Non-DBA’s in charge of DB’s


-3-

• System Administrators

• Web Administrators

• Developers

You get stretched


-4-

WHAT CAN YOU DO IN 5 MINUTES?The top tips to ensure your database is running smoothly


-5-

BIG DISCLAIMER!


-6-

• Its important to distinguish this up front:– I am giving you the easy answers, which are not always

the correct answers..

– If you only have 5 minutes to spend on one of these

database topics, you lose a lot of flexibility

– I could spend lots of time talking about everything here,

instead I am focusing on maximizing your benefit.

• This presentation assumes you are running on Linux/Unix

The Common Problems Kill You


-8-

THOU SHALL HAVE BACKUPS!


-9-

They can save your bacon!

• Probably the most common problem to get called on is some sort of slowness

• Keys to success– Identify what is slow

– When its slow

– Can you repeat it

• Dig in

The Slow Down Of Doom!


-10-

No sense in targeting area’s that do not matter

Target your pain first


-11-

• Google is your friend to research problems• But don’t tune via Google• Trust but verify

Know where to look


-12-

• Know the tools out in the community• Have them ready to jump in and fight fires at a

moments notice• Tools like:

– Percona Toolkit

– Cacti

– Home grown scripts

– etc

Know thy tools


-13-

• Define what is slow? If its a single page, the process is drastically different then if its general slowness.

• Assume General Slowness– #1 What has changed? A new deployment? More users?

Etc? Sometimes a roll-back of a buggy code release is

the best fix.

• DO NOT ASSUME ITS THE DATABASE!• Start with the obvious and the quick and move to

more complex

Tuning walk through


-14-

• 5.1 -vs- 5.0 provides improvements

• 5.5 -vs- 5.1 provides improvements

• 5.6 –vs- 5.5 provides improvements

HUGE Differences in Versions!


-15-

• Often stock OS repos are out of date!• Watch out for old versions installed as

dependencies of off the shelf software• Just because a hosting provider installed it does not

mean its the latest or best

Where did you get it?


-16-

• Innodb• Innodb Plugin• XtraDB• TokuDB• MyISAM• Aria• Many Many more....

Storage Engines


-17-

• Old Kernel?• Multi-core servers, scalability getting better (~32 cores + )• Fit your hot data into memory!

– Typically more memory is better

• IO is a Major bottleneck– RAID For your databases is a must

– Battery Backed Controller

– RAID 10 typically better then RAID 5

– SSD's for optimal performance

• Network should not be forgotten• 32bit binary or OS on 64 bit machine

Hardware & OS


-18-

• Next I start by looking at the server (Sar, vmstat, top, iostat -x)... I typically start low and work my way into the database– Too much io, maybe not enough memory, missing

indexes, or bad config– Cpu maxed out could be lots of users, lots of concurrent

queries, or something else.– Is the problem even MySQL? Often times we see issues

with non-database software– top process is something else ( i.e. apache, php, cron

job, etc )– Network is often overlooked

Review OS Metrics


-19-

COMMON TOOLSFor monitoring the OS/MySQL in a pinch


-20-

• Sar is your friend, every unix/linux server should setup sar to collect stats in the background– Sar Can give you historical reports on : CPU, Disk,

Memory, Network, and a lot more.

– Scheduled to collect data via cron

• Iostat is valuable in gauging disk performance– avwait+svctim = response time (iostat -x)

Systat


-21-

Sar Network Stats


-22-

Sar CPU


-23-

Sar Memory


-24-

Iostat


-25-

VMSTAT


-26-

• Can give you a realtime view of changes on the system to CPU/Memory

Top


-27-

• Get a current Snapshot of running processes

Percona Toolkit


-28-

• Collection of scripts used for monitoring system and DB performance

• pt-stalk• pt-diskstats• pt-pmp• Pt-Ioprofiler• pt-mext

Cacti


-29-

• Great Visual Presentation of Historical Stats• MySQL, Apache, Sphinx, Memcached, (Entire

Stack)• Percona Maintains the MySQL Templates• Can add a lot of value...

Example:

Mysterious Slowdown

OMG The Server is Slow!


-30-

Dig a little deeper


-31-

Oh Wait it Always Happens


-32-

Percona Monitoring Plugins- Cacti!


-33-

Grab them here: http://www.percona.com/software/percona-monitoring-plugins/

http://www.percona.com/software/percona-monitoring-plugins/

THE CONFIGURATIONReasonability Check


-34-

MYSQL CHECKSLooking at the DB


-35-

Configuration Options – Top 10


-36-

General Configuration Options• query_cache_size (Disable in many versions)• log_slow_queries• long_query_time• max_connections• expire_logs_days

MyISAM Specific• key_buffer_size

Configuration Options


-37-

Innodb Specific Options• innodb_buffer_pool_size• innodb_flush_log_at_trx_commit• innodb_log_file_size

Plugin/Xtradb/Mysql 5.5 Innodb Options• innodb_io_capacity• innodb_read_ahead• innodb_write_io_threads• innodb_read_io_threads

Online Tools to help you set your my.cnf!


-38-

• https://tools.percona.com/wizard

Memory and Swap


-39-

• Disk IO is bad, Avoiding swapping things that should be in memory to disk ( do not disable swap however )

• SSD’s/Fast Disks• Hot Data in Memory!• read_buffer, read_rnd_buffer, sort_buffer, join_buffer

are all allocated per thread so be careful!• You want temp tables to be built in memory not on disk

Check Current Activity

• Mysql

– “show full processlist”

– “show global status”

– “Show innodb status”

• Mysqladmin

• Slow Query Log

• pt-query-digest

• Mysqldumpslow

• Innotop

• Mytop

• pt-mext


-40-

Show Processlist


-41-

mysql> show processlist\G

*************************** 1. row ***************************

Id: 52

User: bench

Host: domU-12-31-39-0C-41-81.compute-1.internal:42315

db: simple_benchmark

Command: Sleep

Time: 0

State:

Info: NULL

*************************** 14. row ***************************

Id: 65

User: bench



Command: Query

Time: 0

State: init

Info: insert into simple_benchmark.simple_comments (lookup_id, comment, mydate) values ( '8519163', 'Comme

*************************** 15. row ***************************

Id: 66

User: bench



Command: Sleep

Time: 0

State:

Info: NULL

Mysqladmin


-42-

• Mysqladmin extended -r -i1 will give you mysql stats at 1 second intervals, change the -i to increase decrease...

• A few stats of note:

pt-mext


-43-

• Mext can take the output from mysqladmin and put the status variables side by side, this makes it very convenient to look for differences

Slow Query Log


-44-

• The Slow Query log is your friend! As mentioned earlier

queries that take over a certain amount of time are

logged here. (Percona Patches for complete picture )

• Summarize the slow query log with percona toolkit, pt-

query-digest or mysqldumpslow if the percona toolkit is

not available

• Look for not only the longest running query, but also the

one with lots of executions and cumulated time

Pt-query-digest output


-45-

pt-query-digest


-46-

Memcached/Redis/Other Caching


-47-

• Review: More memory the better

• Caching frequently used data is an excellent way to

boost performance

• This typically requires changes to your app

• Performance improvement can be dramatic

Query Tuning


-48-

• Review the slow query log, parse the log with pt-

query-digest.

– If I see something here, I will analyze it

• If your still at a loss, try using tcpdump to dump

SQL statements into pt-query-digest and look for

repeated queries. ( Easier then the general query

log if you need sub second slow queries).

Query Tuning


-49-

• Use Explain Plan

• Add Indexes where needed, do not over index

– pt-duplicate-key-checker

• Lots of small queries can sometimes be more

impactful than 1 big one.

Remember


-50-

• The Disclaimer!

• Questions?

The 5 Minute DBA-DBA Skills for Non-DBA

Software