Asynchronous Replication in MySQL (and what came after it)

MySQL 5.6 Replication for AdminsKristian Köhntopp

How async replication works2

Master Slave

Binlog RelayLog

ConnectionThread

IOThread

SQLThread

To Tables

Slavelogs in to Master

Server ID: x Server ID: y

Setting up a master

• --server_id = x

• --binlog_format = STATEMENT|ROW|MIXED

• --expire_logs_days = n

• --log_bin = name -- Restart required

• SHOW MASTER STATUS

• SHOW MASTER LOGS

3

Making a full backup

• Slave start position:

• A consistent backup associated with a binlog position.

• mysqldump --master-data=2 OR

• mylvmbackup

4

Setting up a slave

• --server-id= y

• Recover from backup.

• Slave now at backups binlog position.

• CHANGE MASTER TOMASTER_HOST, MASTER_PORT,MASTER_USER, MASTER_PASSWORD MASTER_LOG_FILE, MASTER_LOG_POS

5

Starting the slave

• SHOW SLAVE STATUS\G

• START SLAVE IO_THREAD; • SHOW SLAVE STATUS\G -- error.log

• START SLAVE SQL_THREAD; • SHOW SLAVE STATUS\G

6

Replication stuck

• set global sql_slave_skip_counter = 1;

• start slave;

• But you need to understand what happened.

• Percona Toolkit:pt-table-sync,pt-table-check

7

Binlog Management

• max-binlog-size

• FLUSH LOGS

• SHOW MASTER LOGS

• PURGE MASTER LOGS BEFOREnow() - INTERVAL 3 DAY

• expire_logs_days

8

Debugging Replication

• mysqlbinlog --help

• --start-datetime, --stop-datetime

• useful only for zooming in, time is a binlog position!

• --verbose is not what you think (-v -vv)

• BINLOG statement decoder

9

Filtering the Binlog

• Do not use binlog-do/ignore-db

• incomplete binlog = no roll forward

• Do not use replicate-do/ignore-db, replicate-do/ignore-table.

• Do use replicate-wild-do/ignore-table.

• Do not mix do and ignore, if at all possible.

10

Unslaving

• STOP SLAVE

• maybe START SLAVE UNTIL MASTER_LOG_FILE = …, MASTER_LOG_POS = …

• RESET SLAVE

11

Files

• On the master:

• binlog.index, binlog.nnnnnn

• On the slave:

• master.info

• relay.info

• relay.nnnnnn

12

Binlog Formats: SBR vs. RBR

• “There is only SBR.”

• binlog_format = ROW|MIXED|STATEMENT generates: BINLOG “<base64>”contains: before-image, after-image

• undebuggable, so 5.6 adds binlog_rows_query_log_events = 1

13

RBR Binlog Sizes

• Typical usage: RBR is 33% to 50% size of SBR.

• RBR size escalates when BLOB types are part of replication.

• max_allowed_packet problem (when FULL is used) slave-max-allowed-packet= (2x normal size)

• 5.6: binlog_row_image = FULL|NOBLOB|MINIMAL

14

Alternate RBR uses

• RBR decoding with “mysqlbinlog -v -v”

• strict format, easily parseable, complete list of changes to the server

• Good for change auditing, change extraction.

• Low overhead, avoids triggers.

15

Reliability

• Starting with 5.6:

• Binlog CRC32 Checksums included, never checked.

• master_verify_checksum = 1

• slave_sql_verify_checksum=1

• Always: sync_binlog = 1

• fast, when group commit is on.

16

Reliability

• Slave using tables: fast w/ group commit.

• master-info-repository=TABLErelay-log-info-repository=TABLE

• Hostnames are part of filenames.

• Makes cloning hard. Configure away!

17

Semi-Synchronous Replication SSR

• enabled & one ssr slave connects

• after commit wait for ack or timeout

• if timeout, async,until SSR catches up and acks

18

Pre-SSR workarounds

• Heartbeat tables

• INSERT INTO heartbeat VALUES (name, timestamp)

• master_pos_wait()

• limited to single level hierarchies

19

SSR background

• it makes it much more likely that fewer transactions will be lost when a master disappears

• it throttles busy clients to run no faster than master-slave networking

• http://www.mysqlperformanceblog.com/2012/01/19/how-does-semisynchronous-mysql-replication-work/

20

SSR enhancement

• Google/Percona/Maria 10 SSR enchangement:

• SSR send first, local commit afterwards

• Slow in Oracle MySQL due to global txn lock, fast is Maria 10

21

SSR Master

• INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';

• show plugins;

• set global rpl_semi_sync_master_enabled = 1; # my.cnf

• set global rpl_semi_sync_master_timeout = 5000;

• # millis, add to my.cnf

22

SSR slave

• INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';

• show plugins;

• set global rpl_semi_sync_slave_enabled = 1; # my.cnf

• stop slave io_thread;

• start slave io_thread;

23

SSR checking

• select @@global.rpl_semi_sync_master_clients;

• select @@global.rpl_semi_sync_master_status;

• select @@global.rpl_semi_sync_slave_status;

24

MariaDB Multisource Replication MSR

• Consolidate multi non-overlapping masters into one slave

• all slave-to-master config and vars named, duplicated

• set @@default_master_connection = …

• https://mariadb.com/kb/en/mariadb/mariadb-documentation/replication-cluster-multi-master/replication/multi-source-replication/

25

MariaDB 10 Parallel Replication

• Record degree of parallelism on master

• Replay in parallel on slave

• min(#sql_thds, master_parallelism)

• Faster slave when master more busy.

• Delay transactions on master to force parallism

• http://kristiannielsen.livejournal.com/18435.html

26

PR config

• slave-parallel-threads = 12

• slave_parallel_max_queued = <buffer in bytes>

• execution is still running in-order

• ordering late in commit stage

27

PR config

• On master, force parallel:

• binlog_commit_wait_count = 100

• binlog_commit_wait_usec = 100000

• Also:

• innodb_flush_logs_at_trx_commit=1

• sync_binlog = 1

28

PR out-of-order execution

• Domain-ID

• application controlled

• set session gtid_domain_id = 1

• alter table t add index i(i)

• set session gtid_domain_id = 0

29

Upcoming technology

• GTID

• SSR + GTID = easier master HA

• Really?

• Galera

• proper “Multi-Master”

• for small values of Multi-Master

30

Upcoming technology

• All new replication technology requires:

• Clean setup, defined as:

• pure InnoDB, full ACID, RBR, clean transactions, QC off, log to file (not table), innodb_autoinc_lock_mode = 2, innodb_doublewrite = 1, no replication filters, sensible txn size (1000-10000 rows, < 50MB), PRIMARY KEYS on all tables

31