MySQL High Availability with Group Replication

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

MySQL High Availabilitywith Group Replication

Nuno Carvalho ([email protected])Principal Software Engineer, MySQL Replication Service Team Lead

1


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2


Program Agenda


Program Agenda

Background

MySQL Group Replication

Architecture

Use cases

Performance

Conclusion

1

2

3

4

5

4

6


Background1


M S

S

S

S

M

write clients read clientsread clients

write clients

Morereads?More

slaves!

Read scale-out

Background: What is Replication Used For?

6


C

B

A

C

B

ACrash

C

B

A

B is thenew master

Uh Oh! Whew!

Redundancy: If master crashes, promote slave to master

Background: What is Replication Used For?

7



• What is MySQL Group Replication?

“Multi-master update everywhere replication plugin for MySQL with built-in automatic distributed recovery, conflict detection and group membership.”

• What does the MySQL Group Replication plugin do for the user?– Removes the need for handling server fail-over.

– Provides fault tolerance.

– Enables update everywhere setups.

– Automates group reconfiguration (handling of crashes, failures, re-connects).

– Provides a highly available replicated database.

8



9

M M M M M

Replication Group

Clients

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 10

Timeline - Server GA and GR Releases

MySQL 5.6.10

MySQL 5.7.9lifecycle interfacesP_S tables for GRServer side changes

GR 0.2.0 labsHello world!

GR 0.3.0 labssupport for corosync 2.x

GR 0.4.0 labsversion handling

GR 0.5.0 labsauto-inc fields handlingrecovery enhancements

GR 0.6.0 labsmulti-platform supportPaxos-based consensus

GR 0.7.0 labsbug fixes

GR 0.8.0 labs (beta)performance enhancementsreplication stream compressionSSL supportIP whitelistingread-only modeerror log enhancementssplit brain handling

GR 0.9.0 labs (RC)multi-threaded applier supportsingle primary mode


MySQL Group Replication2

2.1 Multi-Master


http://mysqlhighavailability.com/mysql-group-replication-transaction-life-cycle-explained/

Multi-Master update everywhere!

• Any two transactions on different servers can write to the same tuple.

• Conflicts will be detected and dealt with.

– First committer wins rule.

12

M M M M M

UPDATE t1 SET a=4 WHERE a=2UPDATE t1 SET a=3 WHERE a=1







13

M M M M M

UPDATE t1 SET a=4 WHERE a=2UPDATE t1 SET a=3 WHERE a=1 OKOK







14

M M M M M

UPDATE t1 SET a=2 WHERE a=1UPDATE t1 SET a=3 WHERE a=1







15

M M M M M

UPDATE t1 SET a=2 WHERE a=1UPDATE t1 SET a=3 WHERE a=1 OK



2.1

2.2

Multi-Master

Automatic distributed server recovery


http://mysqlhighavailability.com/distributed-recovery-behind-the-scenes/

Automatic distributed server recovery!

• Server that joins the group will automatically synchronize with the others.

17

M M M M M N

I want to play with you





18

M M M M M N

ONLINERECOVERING





19

M M M M M N

ONLINE




• If a server leaves the group, the others will automatically be informed.

20

M M M M M M

My machine needs maintenanceor a system crash happens

Each membership configurationis identified by a view_id

view_id: 4




• If a server leaves the group, the others will automatically be informed.

21

M M M M M




• Server that (re)joins the group will automatically synchronize with the others.

22

M M M M M M

RECOVERING -> ONLINE

view_id: 5



2.1

2.2

2.3

Multi-Master


MySQL/InnoDB look & feel


http://mysqlhighavailability.com/mysql-group-replication-monitoring/

MySQL look & feel!

• MySQL Plugin

– Regular MySQL Plugin. Nothing new.

• MySQL InnoDB– Use InnoDB as normally you would. Nothing new.

– Transparent optimizations in InnoDB to better support Group Replication.

• MySQL Performance Schema–Monitor Group Replication using regular Performance Schema tables. Nothing new.

24


http://mysqlhighavailability.com/mysql-group-replication-monitoring/

MySQL look & feel!

• Outcome

– Group Replication is no alien component.

– Existing MySQL users feel right at home.

– New MySQL users only have to learn MySQL tech, nothing else.

25


Full GTID support!

• All group members share the same UUID, the group name.

26

M M M M M

INSERT y;Will have GTID: group_name:2

INSERT x;Will have GTID: group_name:1


Full GTID support!

• Users can specify the identifier for the transaction.

27

M M M M M

INSERT y;Will have GTID: group_name:1

SET GTID_NEXT= “UUID:50”INSERT x;


Full GTID support!

• You can even replicate from a outside server to a group, global identifiers will be preserved.

28

M M M M M

Conflicts will be detected!


Full GTID support!

• You can also replicate from a group to a outside server, global identifiers will be preserved.

29

M M M M M



2.1

2.2

2.3

2.4

2.5

Multi-Master



Auto-increment configuration/handling

Full GTID support



• Group is configured not to generate the same auto-increment value on all members.

31

M M M M M

INSERT y;y: 4

INSERT z;z: 11

INSERT x;x: 1


[1]: http://mysqlhighavailability.com/mysql-group-replication-auto-increment-configuration-handling/


• By default, the offset is provided by server_id and increment is 7 [1].

32

M M M M M

INSERT y;y: 4

INSERT z;z: 11

INSERT x;x: 1

server_id: 1server_id: 4



• Users can change the increment size to their needs using GROUP_REPLICATION_AUTO_INCREMENT_INCREMENT option.

33

M M M M M

INSERT y;y: 4

INSERT z;z: 11

INSERT x;x: 1

server_id: 1server_id: 4


Plugin version access control


2.1

2.2

2.3

2.4

2.5

2.6

Multi-Master




Full GTID support


Plugin Version Access Control

• When joining, versions are crucial when determining if a member is compatible with a group.

35

5.7.13 5.7.13 5.7.14 5.7.15 5.7.12

Member with different patchversion is allowed to join




36

8.0.0 8.0.0 8.0.0 8.0.0 5.7.15

Member with lower major versionthan the major version inthe group is not allowed to join




37

5.7.14 5.7.14 5.7.15 5.7.15 8.0.0

Member with higher majorversion is allowed to join butis not allowed to do writes


Built-in communication engine

Plugin version access control


2.1

2.2

2.3

2.4

2.5

2.6

2.7

Multi-Master




Full GTID support


http://mysqlhighavailability.com/mysql-group-replication-for-mysql-5-7-9/

Built-in Communication Engine

• Feature rich new replication plugin based on proven distributed systems algorithms (Paxos).

– Compression, multi-platform, dynamic membership, distributed agreement, quorum based message passing, SSL, IP whitelisting.

• No third-party software required.

• No network multicast support required.

–MySQL Group Replication can operate on cloud based installations on which multicast is not allowed.

39


Read-only mode


2.8


http://mysqlhighavailability.com/mysql-group-replication-important-behavior-and-configuration-changes/

Read-only mode

• When a member joins the group, during distributed recovery, read-only mode is set automatically.

• On a unlikely event of a member failure, read-only mode is set automatically to prevent inconsistency with the group and member state changes to ERROR.

41

M M M M M


Full stack secure connections

Read-only mode


2.8

2.9


http://mysqlhighavailability.com/mysql-group-replication-securing-the-perimeter/


• Following the industry standards, Group Replication supports secure connections along the complete stack

– Client connections

– Distributed recovery connections

– Connections between members

• IP Whitelisting

– Restrict which hosts are allowed to connect to the group

– By default it is set to the value AUTOMATIC, which allows connections from private subnetworks active on the host

43


Parallel applier support


Read-only mode


2.8

2.9

2.10



• Group Replication now also takes full advantage of parallel binary log applier infrastructure

– Reduces applier lag and improves replication performance considerably

– Configured in the same way as in asynchronous replication

45

--slave_parallel_workers=NUMBER--slave_parallel_type=logical_clock--slave_preserve_commit_order=ON


Single Primary Mode



Read-only mode


2.8

2.9

2.10

2.11



Single Primary Mode

• Configuration mode that makes a single member act as a writeable master (PRIMARY) and the rest of the members act as hot-standbys (SECONDARY).

– The group itself coordinates automatically to figure out which is the member that will act as the PRIMARY, through a leader election mechanism.

• Default mode

– Closer to classic asynchronous replication setups, simpler to reason about from the beginning.

– Avoids some of the limitations of multi-master mode.

47



Single Primary Mode

• Automatic leader election mechanism

• Secundaries are automatically set to read-only

48

S S S S P S

Primary



Single Primary Mode

• Automatic leader election mechanism

49

S S S P S

Primary



Single Primary Mode

• The current primary member UUID can be known by executing the following SQL statement

50

mysql> SELECT * FROM performance_schema.global_status WHERE VARIABLE_NAME='group_replication_primary_member';VARIABLE_NAME VARIABLE_VALUEgroup_replication_primary_member dcd3b36b-79c5-11e6-97b8-00212844d44e


Requirements

Single Primary Mode



Read-only mode


2.8

2.9

2.10

2.11

2.12


• Concurrent DDL (on multi-master)

Requirements (by design)

• Requires InnoDB storage engine

• Primary key is required on every table

• Requires global transaction identifiers turned on

• Requires binary log turned on

• Requires binary log row format

• Optimistic execution: transactions may abort on COMMIT due to conflicts with concurrent transactions on other members

• Up to 9 servers in the group

52

• Serializable (on multi-master)

• Cascading Foreign Keys (on multi-master)

• Transaction savepoints

• Binary log events checksum

Forbidden

Warnings


ArchitectureIntroduction

3

3.1


MySQL Group Replication is

• Built on top of proven technology!

– Shares many pieces of MySQL Replication.

–Multi-Master approach to replication.

• Built on reusable components!– Layered implementation approach.

– Interface driven development.

– Decoupled from the server core.

– The plugin registers as listener to server events.

– Reuses the capture procedure from regular replication.

– Provides further decoupling from the communication infrasctructure.

54


Major Building Blocks


3

3.1

3.2


Major Building Blocks (1)

56

M M M M M

Com. API

ReplicationPlugin

API

MySQLServer

Group Comm.

System (Corosync)Group Com. Engine


Major Building Blocks (2)• Server calls into the plugin through a generic

interface

– (Most of server) internals are hidden from the plugin.

– Some of the semi-sync interfaces were reused. Others were created.

• Plugin interacts with the server through a generic interface

– Replication plugin determines the fate of the commit operation through a well defined server interface.

– The plugin makes use of the relay log infrastructure to inject changes in the receiving server.

57

Com. API

ReplicationPlugin

API

MySQLServer

Group Comm.




• The plugin is responsible for

–Maintaining distributed execution context.

– Detecting conflicts.

– Handling distributed recovery:• Detect membership changes;

• Donate state if needed;

• Collect state if needed.

– Receiving and handling transactions from other members.

– Deciding the fate of on-going transactions.

58

Com. API

ReplicationPlugin

API

MySQLServer

Group Comm.




• The communication API (and bindings) is responsible for:

– Abstracting the underlaying communication engine from the plugin itself.

–Mapping the interface to a specific communication engine.

59

Com. API

ReplicationPlugin

API

MySQLServer

Group Comm.




• The communication engine:

– Variant of Paxos developed at MySQL.

– Building block to provide distributed agreement between servers.

60

Com. API

ReplicationPlugin

API

MySQLServer

Group Comm.



The Complete Stack

Major Building Blocks


3

3.1

3.2

3.3


The Complete Stack

62

API

ReplicationPlugin

API

MySQLServer

Performance Schema Tables: Monitoring

MySQL

APIs: Lifecycle / Capture / Applier

InnoDB

Replication Protocol

Group Com. API

Group Com. Engine

Network

Plu

ginCapture Applier

ConflictsHandler

Group Comm.System (Corosync)

Group Com. EngineGroup Com. Binding

Recovery


Use cases4


Use Cases

• Elastic Replication

– Environments that require a very fluid replication infrastructure, where the number of servers has to grow or shrink dynamically and with as little pain as possible.

64

M M M M M


Use Cases

• Highly Available Shards

– Sharding is a popular approach to achieve write scale-out. Users can use MySQL Group Replication to implement highly available shards. Each shard can map into a Replication Group.

65

M M M M M M M M M M M M M M M


Use Cases

• Alternative to Master-Slave replication

• Single Primary mode provides further automation on such setups

– Automatic PRIMARY/SECONDARY roles assignment

– Automatic new PRIMARY election on PRIMARY failures

– Automatic setup of read/write modes on PRIMARY and SECONDARIES

– Global consistent view of which server is the PRIMARY

66

S S S S P S

Primary


Performance5


Performance

68

5000

10000

15000

20000

25000

30000

2 3 5 7 9

THR

OU

GH

PU

T (T

PS)

NUMBER OF SERVERS IN THE GROUP.

Group Replication Throughput(as perceived by the client application)

Single-master Sustained Single-master Peak Multi-master Sustained

Multi-master Peak Single-server (MySQL 5.7.14)

Replication Performance blogs at: http://mysqlhighavailability.com/category/performance/

Servers9 Dual Xeon E5-2660-v3Enterprise SSD Storage10Gbps Ethernet Network

Client1 Dual Xeon E5-2699-v310Gbps Ethernet NetworkSysbench 0.5 RW workload

Peak ThroughputThe number of transactions that writers can propagate to the group (per second).

Sustained ThroughputThe number of transactions that can be propagated to the group without increasing the replication lag on any member (per second).

http://mysqlhighavailability.com/category/performance/


Performance

69

• On a sustained throughput:

–Multi-master performance degrades gracefully while going from a group with 2 servers to a group with 9 servers.

– Single primary performance degrades marginally when growing the group size.

• On a peak throughput:–Multi-master exhibits 1.8X speedup when compared to the single server.• Read load is balanced across the servers in the group.

• Write load is lower since execution is balanced across the group, whereas in single primary mode the primary becomes a bottleneck.

–With a single primary there is no lag on the other members.


Conclusion6


Summary• Cloud Friendly

– Great techonology for deployments where elasticity is a requirement, such as cloud based infrastructures.

• Integrated

–With server core through a well defined API.

–With GTIDs, row based replication, performance schema tables.

• Autonomic and Operations Friendly

– It is self-healing: no admin overhead for handling server fail-overs.

– Provides fault-tolerance, enables multi-master update everywhere and a dependable MySQL service.

• Lab releases provide a sneak peek at what is coming - a new replication plugin which is the core for the new MySQL InnoDB Cluster.

71


MySQL InnoDB Cluster: The End Goal

72

S1 S2 S3 S4 S…

M

M M

MySQL Connector

Application

MySQL Router

MySQL Connector

Application

MySQL Router

MySQL Shell

HA

Rep

licaS

et1

S1 S2 S3 S4 S…

M

M M

MySQL Connector

Application

MySQL Router

HA

Rep

licaS

et 2

Rep

licaS

et 3

MySQL Connector

Application

MySQL Router

S1 S2 S3 S4

M

M M

HA


Where to go from here?

• Packages

– http://labs.mysql.com

• Blogs from the Engineers (news, technical information, and much more)– http://mysqlhighavailability.com

73

http://labs.mysql.com/

http://mysqlhighavailability.com/