Top Banner
Linux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros
43

Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Feb 06, 2018

Download

Documents

lexuyen
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Linux Internals For MySQL DBAs

Ryan Lowe Marcos Albe Chris Giard

Daniel Nichter Syam Purnam

Emily Slocombe Le Peter Boros

Page 2: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Linux Kernel• It’s big (almost 20 million lines of code)

• It’ll take you YEARS to be an expert

• Resources:

• linuxfromscratch.org

• kernel.org

• man pages

• lwn.net

Page 3: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros
Page 4: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

What do we care about?

• Networking

• Storage

• Memory

• Processing

Page 5: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

What do we care about?

• Storage

• Memory

• Processing

• Networking

Page 6: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

/proc

Page 7: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

/proc• Processes and other system information in a

hierarchical file-like structure

• Interaction between kernel space and user space

• Exposes kernel knobs and sliders

• Plain text files

• echo 'value' > /proc/you/want/adjusted

Page 8: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

/proc• cpuinfo

• meminfo

• interrupts

• diskstats

• fs/

• net/

• sys/

Page 9: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

ulimits

• Considered part of security system

• For performance/operations we only care about open files (-n)

• For debugging might need to set core dump size limit to Unlimited (-c)

Page 10: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Storage

Page 11: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Virtual File System (VFS)

• Abstraction layer to allow Linux to handle many filesystems.

• Provides a common interface to make your life easier.

• Introduces the Common File Model

Page 12: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Common File Model

• Superblock

• Inode

• File

• Dentry

Page 13: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Filesystems• Btrfs

• ext2/3/4

• ReiserFS

• XFS

• ZFS

Page 14: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

ext2/3/4• ext2: Old; No Journaling (SSD/Flash, maybe)

• ext3: ext2 + journaling + HTree Indexing

• ext4

• Large Volumes

• Extents

• Checksummed Journal

• Nanosecond Timestamps

Page 15: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

XFS

• Standard recommendation for DB workloads

• Highly proficient with parallel IO

• Current (started 1998, but development is active)

Page 16: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

AIO

A method for performing IO operations so that the process that issued an IO request is not blocked till

the data is available.

Instead, after an IO request is submitted, the process continues to execute its code and can later check the

status of the submitted request.

Page 17: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Mount Options• no[dir]atime

• nobarrier

• discard

!

%> mount -o remount,rw,new,options,here

Page 18: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

IO Schedulers• noop

• FIFO Queue w/Request Merging

• deadline

• Impose a deadline on all operations

• cfq

• Completely Fair Queueing

• anticipatory

• “Anticipates” synchronous read operations

Page 19: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Storage: Disks• Rotational Latency: The delay waiting for the rotation of

the disk to bring the required disk sector under the read-write head.

• Seek time: Time to move the Read/Write Head from current position to the desired track location

• Access time (Response time): How fast we can locate a position of a file

• Transfer time (Throughput): How fast we can get bytes from disk to RAM

Page 20: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

IOPS

1 / (average latency in ms + average seek time in ms)

Page 21: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

IOPS Example• Model: Western Digital VelociRaptor 2.5" SATA hard

drive

• Rotational speed: 10,000 RPM

• Average latency: 3 ms (0.003 seconds)

• Average seek time: 4.2 (r)/4.7 (w) = 4.45 ms (0.0045 seconds)

• Calculated IOPS for this disk: 1/(0.003 + 0.0045) = about 133 IOPS

Page 22: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

RAID

• Hardware RAID w/BBU (or NVRAM)

• RAID 0, 1, 5, 6, 10, 50, 60, etc

Page 23: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

RAID Alignment

Page 24: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

RAID Alignment

http://www.mysqlperformanceblog.com/2011/06/09/aligning-io-on-a-hard-disk-raid-the-theory/

Page 25: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

InnoDB Flush Method

• fdatasync

• O_DSYNC

• O_DIRECT

• O_DIRECT_NO_FSYNC

Page 26: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Memory

Page 27: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Memory: Swap

Page 28: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros
Page 29: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

InnoDB Buffer Pool Size

• Allocated Dynamically

• unless you use innodb_buffer_pool_populate

• InnoDB checks on boot if enough RAM is available

• 10% overhead

Page 30: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Huge Pages• Translation Lookaside Buffer (TLB)

• Default 4k page size

• Larger pages = Smaller TLB

• Huge Pages:

• 2MB - commodity HW

• 1GB - High End (1TB+)

• Huge Pages are good for huge Buffer Pool

Page 31: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

NUMA• Each physical die is a NUMA node

• introspection: numactl --hardware / numactl --show

O(log(N))O(1) for local

O(log(N)) for remote

Page 32: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

NUMA• Disable MySQL NUMA affinity / pinning except:

• When you run multiple mysqld instances on the same system

• AND you have data for each instance separated on different PCIe cards

• AND the PCIe cards are local to different CPU sockets

• Percona implements Jeremy Cole’s suggestions from http://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-mysql/

Page 33: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

malloc

Page 34: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Processing

Page 35: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

CPU Time

Page 36: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Interrupts

Page 37: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Context Switching

Page 38: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

CPU Governors• Control CPU speed and power consumption

• # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

• control through /proc (or /etc/init.d/cpuspeed or cpupower)

• More: http://forum.xda-developers.com/showthread.php?t=1736168

• Most distros use on-demand by default (/etc/defaults/cpupower)

• We care about on-demand, performance, powersave. We usually want performance;

Page 39: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Networking

Page 40: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Backlog• Queue for new TCP connections

• MySQL: back_log

• Linux: tcp_max_syn_backlog

• sysctl -w net.ipv4.tcp_max_syn_backlog = 4096

• sysctl -w net.core.somaxconn = 1024

Page 41: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Slow Links• /proc/net/core/wmem_max = 104857600

• /proc/net/core/rmem_max = 104857600

• /proc/net/ipv4/tcp_wmem = 4096   86400   66060288

• /proc/net/ipv4/tcp_rmem = 8192    86400   66060288

• /proc/net/ipv4/tcp_mem  = 104857600 104857600 104857600

• /proc/net/ipv4/tcp_window_scaling = 1

• /proc/net/ipv4/tcp_sack           = 1

• /proc/net/ipv4/tcp_timestamps     = 1

• /proc/net/ipv4/tcp_no_metrics_save = 0

• /proc/net/ipv4/tcp_moderate_rcvbuf = 1

Page 42: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

High QPS• /proc/net/ipv4/ip_local_port_range = 15000 61000

• /proc/net/ipv4/tcp_max_tw_buckets = 2000000

• /proc/net/ipv4/tcp_tw_reuse = 1

• /proc/net/ipv4/tcp_syncookies = 1

• /proc/net/core/wmem_default = 135168

• /proc/net/core/rmem_default = 135168

• /proc/net/ipv4/tcp_wmem = 4096 86384 104857600

• /proc/net/ipv4/tcp_rmem = 8192 86384 104857600

• /proc/net/ipv4/tcp_mem = 104857600 104857600 104857600

• /proc/net/core/rmem_max = 104857600

• /proc/net/core/wmem_max = 104857600

Page 43: Linux Internals for MySQL DBAs - Percona · PDF fileLinux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Questions