Copyright 2016 Severalnines AB 1 Your host & some logistics I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: [email protected] during or after the webinar.
35
Embed
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Performance Tuning
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Copyright 2016 Severalnines AB
1
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar!
Feel free to ask any questions in the Questions section of this application or via the Chat box.
You can also contact me directly via the chat box or via email: [email protected] during or after the webinar.
! Never-ending story which starts once you install MySQL on the host
! You have to tune for a specific workload
! Workload may change in time
! More data can make it I/O-bound
! Different query mix may increase CPU load and put stress on different parts of the InnoDB
! Keep in mind that configuration tuning is not likely to give you a huge increase in performance (except if the server is really badly configured)
Copyright 2016 Severalnines AB
8
Tuning process
Copyright 2016 Severalnines AB
9
Tuning process
! You need a deterministic, test environment to make sure you can measure the impact of the changes
! Environment should mirror production as close as possible, to make it more relevant
! Changes should be introduced one at a time to ensure you understand the impact of each of them
! Benchmark the system using queries as close to production as possible
! Restore it to the original state for another round of tweaking
! Rinse and repeat until you are happy with results
Copyright 2016 Severalnines AB
10
Tuning process
! Grab a backup of your production systems
! Restore it on a host
! Capture real-world queries using slow log or tcpdump
! Make _one_ change in my.cnf or OS settings
! Restart MySQL or reboot the host itself
! Replay queries using Percona Playback or pt-upgrade
! Measure the difference, repeat the process by restoring the backup if you want to make one more change
Copyright 2016 Severalnines AB
11
Tuning the operating system for MySQL
Copyright 2016 Severalnines AB
12
Tuning the operating system for MySQL - memory
! Minimize the flushing operations in OS to ensure no stalls are happening when dirty data is flushed to disk
! vm.dirty_ratio - a hard limit of the memory that can be used to cache dirty disk pages
! vm.dirty_background_ratio - percentage of system memory that can be used to cache modified (“dirty”) pages before the background flush process kicks in
! We want them to be set to low numbers - 5 - 10% to trigger flushing often and make it quick
! Default settings may result in a situation where 40% of your memory has to be flushed to disk at once
! For large instances (128GB) it can be even >50GB of data to flush
Copyright 2016 Severalnines AB
13
Tuning the operating system for MySQL - memory
! Swapping - a process of moving data from memory to disk
! MySQL doesn’t like active swapping - we want data in memory, not on disk
! but we even more don’t want to be killed when system runs out of memory
! vm.swappiness = 1 - starting from kernel 3.5 this is the setting you want to use - allow some swapping but only when there’s no other option
! You can also deprioritize MySQL when it comes to OOM Killer:
! echo -1000 > /proc/[pid]/oom_score_adj
! should help in environments shared with other processes
Copyright 2016 Severalnines AB
14
Tuning the operating system for MySQL - memory
! When running MySQL on bare iron, you need to make sure you have NUMA interleave policy set
! Starting from 5.6.27 a variable is available: innodb_numa_interleave
! Before that you could run MySQL as:
! numactl --interleave=all $command
! Memory allocation library may impact performance of MySQL
! Many choices: glibc malloc, tcmalloc and jemalloc
! Use LD_PRELOAD or malloc-lib variable in [mysqld_safe] section of my.cnf
Copyright 2016 Severalnines AB
15
Tuning the operating system for MySQL - memory
Copyright 2016 Severalnines AB
16
Tuning the operating system for MySQL - I/O
! Disk schedulers may impact performance of I/O-bound workloads
! CFQ is tuned for desktop workloads
! Noop or deadline are usually better for MySQL
! Benchmark to see which one of them will work better for you
! Filesystem - most often you’ll use EXT4 or XFS
! Performance depends on workload, hardware and kernel version
! Benchmark to see which one will work better in your environment
Copyright 2016 Severalnines AB
17
Tuning MySQL configuration
Copyright 2016 Severalnines AB
18
Tuning MySQL configuration - memory! InnoDB buffer pool - used to cache data and
store dirty pages
! More is better but you need to leave some memory for other buffers
! Per join buffers
! Per session buffers
! Temporary tables
! You may have heard about 80% rule
! It’s more like 90% for large (i.e. 128GB) hosts
! Make sure you err on the side of ‘too small’
! Unless you run MySQL 5.7 where you can resize InnoDB buffer pool dynamically, without restart
! For fairly loaded (~20-30 running threads) host with 128GB of memory it should be ok to leave ~15GB of memory free
! All depends on the workload so your mileage may vary
Copyright 2016 Severalnines AB
19
Tuning MySQL configuration - memory! Per-session buffers in InnoDB:
! Those options can help you to reduce contention on some of those structures
! Increase number of buffer pools or adaptive hash index partitions if you notice a congestion on them
! Or, preemptively, if you have to handle highly concurrent traffic
! Don’t use buffer pool instances smaller than 1GB (use 2GB+, more instances can slow down the system)
Copyright 2016 Severalnines AB
! max_connections - keep it large enough to handle incoming connections but avoid hundreds of open connections
! If you need to handle thousands of connections, check the connection pooling options or a proxy, ideally with connection multiplexing
! log_bin - you want to have binlogs enabled
! sync_binlog=0 for MySQL 5.7 - it used to be a default on MySQL 5.6, it has been changed in 5.7.7
! skip_name_resolve - just to make sure your database won’t suffer when DNS will not be reachable
! Query cache - optimize it away by disabling
! Use external caching layer (Redis, Memcached)
25
Tuning MySQL configuration
Copyright 2016 Severalnines AB
26
Tools useful in tuning the configuration
Copyright 2016 Severalnines AB
27
Tools useful in tuning the configuration
! Percona developed two great tools, part of the Percona Toolkit suite, which helps you to collect useful data about MySQL and underlying operating system
! pt-summary collects data about hardware and OS settings
! CPU, memory, disk, load on the system
! pt-mysql-summary connects to MySQL instance and gives you data about workload and configuration
! Type of queries that are running, handlers, configuration
! snapshot of current traffic
! and many others
Copyright 2016 Severalnines AB
28
Tools useful in tuning the configuration - pt-summary
Copyright 2016 Severalnines AB
29
Tools useful in tuning the configuration - pt-mysql-summary
Copyright 2016 Severalnines AB
30
Tools useful in tuning the configuration - pt-mysql-summary
Copyright 2016 Severalnines AB
31
What to avoid when tuning MySQL?
Copyright 2016 Severalnines AB
32
What to avoid when tuning MySQL?
! Tuning MySQL is not a simple task - it’s very time-consuming to go through the process of change - test - change - test…
! People like to simplify things and look for shortcuts
! Auto-tuning tools have been created
! mysqltuner.pl
! Configurators and wizards have been created (i.e. by Percona)
! Monitoring and management tools offer advisors (i.e. ClusterControl)
Copyright 2016 Severalnines AB
33
What to avoid when tuning MySQL?
! Some of those tools base on counters from MySQL internal status and this is tricky
! The fact you have joins performed without indices doesn’t justify increasing join_buffer_size
! Sometimes making a good change is a bad decision
! If you run MySQL without binlog enabled, without double_buffer or without XA enabled, you may see a suggestion to enable those features
! Binlog is great for point-in-time recovery
! double_buffer and XA helps keep your data safe
! Enabling them will add serious I/O load on the system - it may not handle it
Copyright 2016 Severalnines AB
34
What to keep in mind when tuning MySQL?
! Make sure you follow the tuning process as we described - one change at a time
! If you use external tools that provide suggestions, take them with a grain of salt
! How competent is the vendor in databases?
! Even if the vendor has the competence - do I understand what this change is and how it may impact my system?
! If you setup a new host, you may use external tools more freely
! This would be initial tuning, not changing the way how application already works
! Make sure you use common sense and each of your changes has a reason