Multi-Master, Multi-Site MySQL Databases Made Easy with ...

©Continuent 2012.

Multi-Master, Multi-Site MySQL Databases Made Easy

with Continuent TungstenRobert Hodges, CEO

Giuseppe Maxia, Director of QA

©Continuent 2012

Introducing Continuent

2

• The leading provider of clustering and replication for open source DBMS

• Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance scaling and data management for MySQL

• Replication - Flexible, high-performance data movement

©Continuent 2012

Quick Continuent Facts

• Largest Tungsten clustering installation processes about 500M transactions each day

• Tungsten Replicator was application of the year at the 2011 MySQL User Conference

• Continuent is well known in the MySQL community for its clustering and replication

• Over 90 customers currently deployed

3

©Continuent 2012

Select Continuent Customers

4

©Continuent 2012 5

Why Tungsten?

©Continuent 2012

MySQL is a Great OLTP Database

6

But how do I...

• Handle DBMS failures?

• Perform maintenance and upgrades without taking applications o!ine?

• Load balance SQL across replicas?

• Create a disaster recovery site?

• Set up multi-master replication?

• Integrate with another DBMS?

©Continuent 2012 7

Tungsten Master/Slave Clustering

©Continuent 2012

Introducing Tungsten Clustering

8

Tungsten clusters combine o"-the-shelf open source DBMS servers into data services with:

• 24x7 data access • Scaling of load on replicas• Simple management commands

...without app changes or data migration

AmazonUS West

apache/php

GonzoPortal.com

Connector Connector

©Continuent 2012

Doesn’t Replication Do This Already?

9

Application

Libmysqlclient.a

Application

Libmysqlclient.a

Replicator

Db2

Replicator

Db3Db1

Replicator

Slave SlaveMaster

©Continuent 2012

Replication Limitations

Replication alone cannot enable constant access to data...

• What happens if the master fails?

• What happens if a slave is lagging too far behind to be usable?

• How do I maintain the DBMS without stopping apps?

10

©Continuent 2012

From Replicating to Clustering

11

Application

Tungsten Connector

Application

Tungsten Connector

Replicator

Db2

Replicator

Db3Db1

Replicator

Slave SlaveMaster

Manager Manager Manager

Monitoring and Control

Monitoring and Control

Data Service: sjc

©Continuent 2012

Basic Cluster Operations

12

• Switch -- Promote a slave node to master

• Shun -- Kick node out of cluster

• Welcome -- Return node to cluster

• Backup -- Run a backup

• Restore -- [Re-]provision a node

• Recover -- Return a broken node to slave

• Heartbeat -- Check cluster liveness

©Continuent 2012

Automatic Failover

13

Application

Tungsten Connector

©Continuent 2012

Automatic Failover

14

Application

Tungsten Connector

1. Detect non-responsivemaster

2. Halt master connections

©Continuent 2012

Automatic Failover

15

Application

Tungsten Connector

3. Select most advanced slave and wait for pending transactions to apply

©Continuent 2012

Automatic Failover

16

Application

Tungsten Connector

4. Promote to master, reconfigure slave(s), and re-enable connections

©Continuent 2012

Automatic Failover

17

Application

Tungsten Connector

5. Recover failed master

©Continuent 2012

Rolling SQL Schema Upgrade

18

Final UpgradeSwitchSlave Upgrade Slave Upgrade

• Shun slave 1, run upgrade, and return to cluster. Discard and recover on failure

• Upgrade old master. Cluster schema is now fully upgraded

• Repeat for additional slave(s)

• Switch master to promote an upgraded slave

©Continuent 2012

Backup, Recovery, and Restore

19

• Backups

/cluster  >  datasource  DB1  backup

• Recover

/cluster  >  datasource  DB2  recover

• Restore

/cluster  >  datasource  DB2  restore

• Out-of-box support for mysqldump/XtraBackup; simple integration for others

©Continuent 2012 20

Managing Data Across Multiple Sites

©Continuent 2012

Topics:

• Tungsten Replicator basics

• Disaster recovery sites

• Multi-master topology

• Fan-in

21

©Continuent 2012

Tungsten Replicator Overview

22

Master

(Transactions + Metadata)

Slave

THL

DBMSLogs

Replicator

(Transactions + Metadata)

THLReplicator

Download transactions via network

Apply using JDBC

©Continuent 2012

Tungsten Replication Service

23

Extract Filter Apply

StageExtract Filter Apply

StageExtract Filter Apply

Stage

Pipeline

MasterDBMS

TransactionHistory Log

In-MemoryQueue

SlaveDBMS

©Continuent 2012

Parallel Replication of Sharded Data

24

Extract Filter Apply

StageExtract Filter Apply

StageStage

Pipeline

MasterDBMS

TransactionHistory Log

In-MemoryQueue

SlaveDBMS

Extract Filter ApplyExtract Filter ApplyExtract Filter Apply

©Continuent 2012

Multiple Services Per Replicator

25

NYCReplicator

London

Service nyc

FrankfurtReplicator

Service fra

Replicator

Service nyc

Service fra

©Continuent 2012

Multi-Master Replication

26

• Updates on 2+ sites (active-active mode)

• Enables geographic distribution of data

• No failover necessary if network fails or site becomes unavailable

• Not all applications can handle multi-master

• Applications must avoid con#icts

• Careful testing required

• Restoration of broken systems may not be easy

©Continuent 2012

Simple Multi-Master Con$guration

27

NYC FrankfurtReplicator

fra (master)

nyc (slave)

Replicator

fra (slave)

nyc (master)

Database-to-Database

©Continuent 2012

Clustered Multi-Master Con$guration

28

NYC Frankfurt

master

slave fra

fra

nyc

nyc

master

slave

©Continuent 2012

Clustered Multi-Master Con$guration

29

NYC Frankfurt

slave

master fra

fra

nyc

nyc

master

slave

NYC Master Offline

©Continuent 2012

Clustered Multi-Master Con$guration

30

NYC Frankfurt

master

slave fra

fra

nyc

nyc

master

slave

NYC Replicator Offline

©Continuent 2012

Complex Multi-Master Topologies

31

StarReplication

AllMasters

©Continuent 2012

Disaster Recovery Sites

32

• Updates go to primary site master

• DR site receives updates in real time

• Simple command promotion

• Tungsten connectors can route to primary and DR sites

• No application changes required

• Simplest way to protect against region/site failures

©Continuent 2012

Tungsten Composite Data Services

33

App Server+ Connector

App Server+ Connector

Master Slave

San Jose New York

sj_nyc

©Continuent 2012

Switching and failover

34

• Planned switch

/sj_nyc  >  switch  to  nyc

• Failover

/cluster  >  failover

• Failover between sites is not automated!

©Continuent 2012

System-of-Record

35

• Application data are “homed” in di"erent locations

• Each MySQL schema has a single master

• Within a single DBMS there can be both masters and slave schemas

• Minimal application changes required

• Allows you to move chunks of data closer to users

©Continuent 2012

System of Record Con$guration

36

NYC Frankfurt

Replicator

fra (master)

nyc (slave)

Replicator

fra (slave)

nyc (master)

acme

spitze

superb

acme

spitze

superb

Acme & Spitze mastered in NYC

Superb mastered in Frankfurt

©Continuent 2012

Implementing System of Record

37

• Con$gure shard $lters on installation to set policies for unknown/bad shards

• De$ne shard homes in shard.map $le

shard_id    master  criticalacme            nyc        falsespitze        nyc        falsesuperb        fra        false

• Load shard de$nitions

trepctl  shard  -­‐insert  <  shard.map

• Move shard homes by editing reloading the shard map

©Continuent 2012

Fan-In Replication

38

• Transactions feed from multiple locations into a single slave

• Aggregate data for reporting

• Consolidate data from multiple servers into one server

• Transfer shards between non-connected clusters/locations

• Con#ict avoidance is up to applications

©Continuent 2012

Fan-In Topology

39

Replicator

Replicator

nyc (master)

fra (master)

sfo (master)

nyc (slave)

fra (slave)

sfo (slave)

Replicator

New York

Frankfurt

Replicator

San Francisco

London

©Continuent 2012

How Do I Obtain Tungsten?

40

• Register at Continuent to get downloads for Oracle replication: http://www.continuent.com/downloads/software

• Documentation is available on Continuent website

• MySQL replication is available from Tungsten Replicator project on code.google.comhttp://code.google.com/p/tungsten-replicator/

©Continuent 2012

One more thing:

WE’RE HIRING!!!

41

©Continuent 2012.

Continuent Web Page:http://www.continuent.com

Tungsten Replicator 2.0:http://code.google.com/p/tungsten-replicator

Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.org/bloghttp://www.continuent.com/news/blogs

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]