©Continuent 2013
Synchronize your data between MySQL
and MongoDB using Tungsten Replicator
Giuseppe Maxia, Director of QAContinuent, Inc
1Tuesday, October 15, 13
©Continuent 2013
About me
2
• Giuseppe Maxia, a.k.a. "The Data Charmer"
• Director of Quality Assurance, Continuent, Inc
• 25+ years development and DB experience
• Long timer MySQL community member.
• Oracle ACE Director
• Blog: http://datacharmer.blogspot.com
• Twitter: @datacharmer
2Tuesday, October 15, 13
©Continuent 2013
Introducing Continuent
3
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
3Tuesday, October 15, 13
©Continuent 2013
A Review of Tungsten Replicator
4
4Tuesday, October 15, 13
©Continuent 2013
Tungsten Replicator Overview
5
Master
(Transactions + Metadata)
Slave
THL
DBMSLogs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
5Tuesday, October 15, 13
©Continuent 2013
Master Replication Service
6
Extract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
MySQLMaster
TransactionHistory Log
In-MemoryQueue
Slave ReplicatorsBinlog
tcp/ip
6Tuesday, October 15, 13
©Continuent 2013
Slave Replication Service
7
Extract Filter Apply
StageExtract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
TransactionHistory Log
In-MemoryQueue
SlaveDBMS
MasterReplicator
tcp/
ip
7Tuesday, October 15, 13
©Continuent 2013
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL MySQL
8Tuesday, October 15, 13
©Continuent 2013
MongoDB in a nutshell
9
9Tuesday, October 15, 13
©Continuent 2013
What is MongoDB
10
• A non-relational database
• A document-oriented database
• Schema-free
• Open source
• High performance
• Scalable
• Developer-friendly (sort of)
10Tuesday, October 15, 13
©Continuent 2013
What is MongoDB good for?
• Storing large amount of unrelated data
• Data that can't be constrained in a schema
• Complement to relational data
• IT IS NOT a drop-in replacement for a relational database
11
11Tuesday, October 15, 13
©Continuent 2013
Relational vs. document
12
personpersonperson
p_id name age
1 Joe 30
2 Fred 23
3 Jack 26
4 Sue 25
5 Pete 32
departmentdepartmentd_id name
1 sales2 dev3 supportpers_deptpers_dept
p_id d_id
1 2
2 2
3 1
4 3
5 1Relational
12Tuesday, October 15, 13
©Continuent 2013
Relational vs. document
13
personpersonperson
p_id name age
1 Joe 30
2 Fred 23
3 Jack 26
4 Sue 25
5 Pete 32
departmentdepartmentd_id name
1 sales2 dev3 supportpers_deptpers_dept
p_id d_id
1 2
2 2
3 1
4 3
5 1Relational
13Tuesday, October 15, 13
©Continuent 2013
Relational vs. document
14
personpersonperson
p_id name age
1 Joe 30
2 Fred 23
3 Jack 26
4 Sue 25
5 Pete 32
departmentdepartmentd_id name
1 sales2 dev3 supportpers_deptpers_dept
p_id d_id
1 2
2 2
3 1
4 3
5 1Relational
14Tuesday, October 15, 13
©Continuent 2013
Relational vs. document
15
personpersonperson
p_id name age
1 Joe 30
2 Fred 23
3 Jack 26
4 Sue 25
5 Pete 32
departmentdepartmentd_id name
1 sales2 dev3 supportpers_deptpers_dept
p_id d_id
1 2
2 2
3 1
4 3
5 1Relational
15Tuesday, October 15, 13
©Continuent 2013
Relational vs. document
16
personpersonperson
p_id name age
1 Joe 30
2 Fred 23
3 Jack 26
4 Sue 25
5 Pete 32
departmentdepartmentd_id name
1 sales2 dev3 supportpers_deptpers_dept
p_id d_id
1 2
2 2
3 1
4 3
5 1Relational
16Tuesday, October 15, 13
©Continuent 2013
Relational vs. document
17
personpersonperson
p_id name age
1 Joe 30
2 Fred 23
3 Jack 26
4 Sue 25
5 Pete 32
departmentdepartmentd_id name
1 sales2 dev3 supportpers_deptpers_dept
p_id d_id
1 2
2 2
3 1
4 3
5 1Relational
17Tuesday, October 15, 13
©Continuent 2013
Relational vs. document
18
personpersonperson
_id name age department
1 Joe 30 dev
2 Fred 23 dev
3 Jack 26 sales
4 Sue 25 support
5 Pete 32 sales
Document
18Tuesday, October 15, 13
©Continuent 2013
How MongoDB keeps data
• three levels:
• dbs
• collections
• documents
19
19Tuesday, October 15, 13
©Continuent 2013
MongoDB insertion demo
> show collections>
> db.person.insert( {_id: 1, name: "Joe", age: 30, department: "dev"})> show collectionspersonsystem.indexes
20
20Tuesday, October 15, 13
©Continuent 2013
MongoDB insertion demo
> db.person.insert( {_id: 2, name: "Fred", age: 23, department: "dev"})> db.person.insert( {_id: 3, name: "Jack", age: 26, department: "sales"})> db.person.insert( {_id: 4, name: "Sue", age: 25, department: "support"})> db.person.insert( {_id: 5, name: "Pete", age: 30, department: "sales"})> db.person.find(){ "_id" : 1, "name" : "Joe", "age" : 30, "department" : "dev" }{ "_id" : 2, "name" : "Fred", "age" : 23, "department" : "dev" }{ "_id" : 3, "name" : "Jack", "age" : 26, "department" : "sales" }{ "_id" : 4, "name" : "Sue", "age" : 25, "department" : "support" }{ "_id" : 5, "name" : "Pete", "age" : 30, "department" : "sales" }
21
21Tuesday, October 15, 13
©Continuent 2013
MySQL to MongoDB basics
22
22Tuesday, October 15, 13
©Continuent 2013
Replication from MySQL to MongoDB
23
• Requires ROW-based-replication
• Replication happens by table
• There is no consolidation into "documents"
• DDL commands are ignored
• Statement commands are ignored
• Column names become document attributes
• enum and set columns are converted to strings
23Tuesday, October 15, 13
©Continuent 2013
First example of replication
# MySQL
create schema oneschema;use oneschema ;create table myfirst( num int not null primary key, dt datetime, ts timestamp, going enum('yes', 'no'));
# MongoDB> show dbslocal 0.078125GBtest 0.203125GBtungsten_mysql2mongodb 0.203125GB
# NOTICE: no "oneschema"
24
24Tuesday, October 15, 13
©Continuent 2013
Inserting data
# MySQLinsert into myfirst values (1, '2003-04-26 09:15:00', null, 'yes');Query OK, 1 row affected (0.01 sec)
select * from myfirst;+-----+---------------------+---------------------+-------+| num | dt | ts | going |+-----+---------------------+---------------------+-------+| 1 | 2003-04-26 09:15:00 | 2013-10-14 19:39:38 | yes |+-----+---------------------+---------------------+-------+1 row in set (0.00 sec)
25
25Tuesday, October 15, 13
©Continuent 2013
Checking results in MongoDB
# MongoDB> show dbslocal 0.078125GBoneschema 0.203125GBtest 0.203125GBtungsten_mysql2mongodb 0.203125GB
> use oneschemaswitched to db oneschema> show collectionsmyfirstsystem.indexes
> db.myfirst.find(){ "_id" : ObjectId("525c2c5af5d9ca820fcee01d"), "num" : "1", "dt" : "2003-04-26 11:15:00.0", "ts" : "2013-10-14 19:39:38.0", "going" : "yes" }
26
26Tuesday, October 15, 13
©Continuent 2013
Another interesting insertion
#MySQLcreate table t1(_id int not null primary key, c char(10));insert into t1 values (1, 'abc');select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc |+-----+------+1 row in set (0.00 sec)
# MongoDB> db.t1.find(){ "_id" : "1", "c" : "abc" }
27
27Tuesday, October 15, 13
©Continuent 2013
More insertions
# MySQLinsert into t1 values (2,'def'), (3,'ghi'), (4,'jkl'), (5, 'mno');Query OK, 4 rows affected (0.01 sec)Records: 4 Duplicates: 0 Warnings: 0
select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc || 2 | def || 3 | ghi || 4 | jkl || 5 | mno |+-----+------+5 rows in set (0.00 sec)
28
28Tuesday, October 15, 13
©Continuent 2013
More insertions
# MongoDB
> db.t1.find(){ "_id" : "1", "c" : "abc" }{ "_id" : "2", "c" : "def" }{ "_id" : "3", "c" : "ghi" }{ "_id" : "4", "c" : "jkl" }{ "_id" : "5", "c" : "mno" }
29
29Tuesday, October 15, 13
©Continuent 2013
Update and delete as seen on master
update t1 set c = 'ZZZ' where _id = 3;Query OK, 1 row affected (0.00 sec)Rows matched: 1 Changed: 1 Warnings: 0
delete from t1 where _id=2;Query OK, 1 row affected (0.00 sec)
select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc || 3 | ZZZ || 4 | jkl || 5 | mno |+-----+------+4 rows in set (0.00 sec)
30
30Tuesday, October 15, 13
©Continuent 2013
e!ects of update and delete on the slave
# MongoDB
> db.t1.find(){ "_id" : "1", "c" : "abc" }{ "_id" : "3", "c" : "ZZZ" }{ "_id" : "4", "c" : "jkl" }{ "_id" : "5", "c" : "mno" }
31
31Tuesday, October 15, 13
©Continuent 2013
Overview of Tungsten installer
32
32Tuesday, October 15, 13
©Continuent 2013
Overview of Installation Process
1. Set up hosts
2. Prepare MySQL replicas
3. Download software
4. Install using tpm
33
Amazon Setup:https://docs.continuent.com/wiki/display/TEDOC/
Preparing+EC2+Servers
33Tuesday, October 15, 13
©Continuent 2013
How tungsten-installer Works for Basic Master/Slave Deployment
34
db1
db2
db3
Staging copy of files
check prereqscopy codeconfigure
34Tuesday, October 15, 13
©Continuent 2013
Tungsten master/slave replication
35
host1 host2
host3
alpha alpha
alpha
THL THL
THL
installer
35Tuesday, October 15, 13
©Continuent 2013
Bi-directional replication
36
host1 host2
alphaalpha
bravobravo
Install all master and slave services on all hosts at once
installer
36Tuesday, October 15, 13
©Continuent 2013
4 nodes all-masters
37
host1
host4
host2
host3
alphaalpha
bravobravo
charlie
charlie
delta
delta
bravobravo
alpha
delta
delta
charlie
charlie
alpha
37Tuesday, October 15, 13
©Continuent 2013
Tungsten security layer
• Tra!c encryption:
• all data in transit (transaction history logs, or THL) is encrypted using SSL
• all administrative tra!c is encrypted with SSL
• Transparent to the user
• Independent of the database server (works also for heterogeneous replication)
38
38Tuesday, October 15, 13
©Continuent 2013
Tungsten replicator without security
39
replicatorservices
host1
host4
host2
host3
alpha
alpha
alpha alpha
master
slave
THL
THL
THLTHL
plain text
39Tuesday, October 15, 13
©Continuent 2013
Tungsten Replicator with security
40
replicatorservices
host1
host4
host2
host3
alpha
alpha
alpha alpha
master
slave
THL
THL
THLTHL
SSL
SSL SSL
SSL
40Tuesday, October 15, 13
©Continuent 2013
host1 host2
host3
alpha alpha
alpha
THL THL
THL
Installing Master/Slave Replication ...
41
41Tuesday, October 15, 13
©Continuent 2013
master/slave using tpm
42
./tools/tpm install alpha \ --topology=master-slave \ --home-directory=/opt/continuent/replicator \ --replication-user=tungsten \ --replication-password=secret \ --master=host1 \ --slaves=host2,host3,host4 \ --start
42Tuesday, October 15, 13
©Continuent 2013
Installing Master/Slave Replication with MongoDB
43
host1
host4
host2
host3
alpha
alpha
alpha
alpha
43Tuesday, October 15, 13
©Continuent 2013
master/slave with MongoDB
44
./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=master-slave \ --master=host1 \ --replication-user=tungsten \ --replication-password=secret \ --slaves=host2,host3,host4 \ --home-directory=$MYSQL_DEPLOY \ --start-and-report
./tools/tpm configure mysql2mongodb \ --hosts=host4 \ --datasource-type=mongodb \ --replication-port=$MONGODB_PORT
./tools/tpm install
44Tuesday, October 15, 13
©Continuent 2013
Installing Fan-In Replication
45
host1
host4
host2
host3
alphabravo
charliebravo
charlie
alpha
45Tuesday, October 15, 13
©Continuent 2013
fan-in using tpm
46
./tools/tpm install many_towns \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=/opt/continuent/replication \ --masters=host1,host2,host3 \ --slaves=host4 \ --master-services=alpha,bravo,charlie \ --topology=fan-in \ --start
46Tuesday, October 15, 13
©Continuent 2013
Installing Fan-In Replication with MongoDB
47
host1
host4
host2
host3
alpha
bravo
charlie
bravo
charlie
alpha
47Tuesday, October 15, 13
©Continuent 2013
fan-in with MongoDB
48
./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=fan-in \ --masters=host1,host2,host3 \ --master-services=alpha,bravo,charlie \ --slaves=host4 \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=$MYSQL_DEPLOY \ --datasource-type=mysql \ --start-and-report
./tools/tpm configure mysql2mongodb \ --hosts=host4 \ --datasource-type=mongodb \ --replication-port=$MONGODB_PORT
./tools/tpm install
48Tuesday, October 15, 13
©Continuent 2013
Install Multi-Master replication
49
host1 host2
host3
alphaalpha
bravobravo
charlie
charlie
bravo
alpha
charlie
49Tuesday, October 15, 13
©Continuent 2013
multi-master using tpm
50
../tools/tpm install musketeers \ --reset \ --topology=all-masters \ --home-directory=/opt/continuent/replicator \ --replication-user=tungsten \ --replication-password=secret \ --masters=host1,host2,host3 \ --master-services=alpha,bravo,charlie \ --start
50Tuesday, October 15, 13
©Continuent 2013
Install Multi-Master replication with Mongodb
51
host1
host4
host2
host3
alphaalpha
bravobravo
charlie
charlie
bravobravo
alpha
charlie
charlie
alpha
51Tuesday, October 15, 13
©Continuent 2013
multi-master with MongoDB
52
./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=all-masters \ --masters=host1,host2,host3 \ --slaves=host1,host2,host3,host4 \ --master-services=alpha,bravo,charlie \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=$MYSQL_DEPLOY \ --datasource-type=mysql \ --start-and-report
./tools/tpm configure mysql2mongodb \ --hosts=host4 \ --datasource-type=mongodb \ --replication-port=$MONGODB_PORT
./tools/tpm install
52Tuesday, October 15, 13
©Continuent 2013
MongoDB or TokuMX
53
• TokuMX is a drop-in replacement for MongoDB
• Open source project, developed by TokuTek
• https://github.com/Tokutek/mongo
• it includes
• better indexing
• row-level locking (MongoDB locks at db level)
• transactions
• better compression
53Tuesday, October 15, 13
©Continuent 2013
DEMO:MongoDB
and multi master installation
54
54Tuesday, October 15, 13
©Continuent 2013
Joining the Community
55
55Tuesday, October 15, 13
©Continuent 2013
Tungsten Replicator is Open Source
56
• Project home:
http://code.google.com/p/tungsten-replicator/
• Log bugs, "nd builds, post in replicator discussion group
• Documentation:
https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Documentation+Home
https://docs.continuent.com/wiki/display/TEDOC/Deploying+MongoDB+Replication
56Tuesday, October 15, 13
©Continuent 2012.
Continuent Web Page:http://www.continuent.com
Tungsten Replicator 2.1:http://code.google.com/p/tungsten-replicator
Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.blogspot.comhttp://www.continuent.com/news/blogs
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]
57Tuesday, October 15, 13