Top Banner

Click here to load reader

Synchronise your data between MySQL and MongoDB

Nov 07, 2014

ReportDownload

Technology

How to start replication between MySQL and MongoDB using Tungsten Replicator, an open source high performance replication engine.

  • 1. Synchronize your data between MySQL and MongoDB using Tungsten Replicator Giuseppe Maxia, Director of QA Continuent, Inc Continuent 2013 Tuesday, October 15, 13 1
  • 2. About me Giuseppe Maxia, a.k.a. "The Data Charmer" Director of Quality Assurance, Continuent, Inc 25+ years development and DB experience Long timer MySQL community member. Oracle ACE Director Blog: http://datacharmer.blogspot.com Twitter: @datacharmer Continuent 2013 Tuesday, October 15, 13 2 2
  • 3. Introducing Continuent The leading provider of clustering and replication for open source DBMS Our Product: Continuent Tungsten Clustering - Commercial-grade HA, performance scaling and data management for MySQL Replication - Flexible, high-performance data movement Continuent 2013 Tuesday, October 15, 13 3 3
  • 4. A Review of Tungsten Replicator Continuent 2013 Tuesday, October 15, 13 4 4
  • 5. Tungsten Replicator Overview Master Replicator Download transactions via network DBMS Logs (Transactions + Metadata) Slave Replicator Apply using JDBC Continuent 2013 Tuesday, October 15, 13 THL THL (Transactions + Metadata) 5 5
  • 6. Master Replication Service Pipeline Stage Extract Filter Stage Apply Extract Filter Apply tcp/ip Binlog MySQL Master Continuent 2013 Tuesday, October 15, 13 Slave Replicators Transaction History Log In-Memory Queue 6 6
  • 7. Slave Replication Service Pipeline Stage Apply Extract Filter Stage Apply Extract Filter Apply tcp/ip Extract Filter Stage Master Replicator Transaction History Log Continuent 2013 Tuesday, October 15, 13 In-Memory Queue Slave DBMS 7 7
  • 8. master-slave MySQL Oracle fan-in slave Oracle MySQL all-masters Heterogeneous Oracle MySQL MySQL Oracle star Continuent 2013 Tuesday, October 15, 13 8
  • 9. MongoDB in a nutshell Continuent 2013 Tuesday, October 15, 13 9 9
  • 10. What is MongoDB Continuent 2013 Tuesday, October 15, 13 A non-relational database A document-oriented database Schema-free Open source High performance Scalable Developer-friendly (sort of) 10 10
  • 11. What is MongoDB good for? Storing large amount of unrelated data IT IS NOT a drop-in replacement for a relational database Continuent 2013 Tuesday, October 15, 13 Data that can't be constrained in a schema Complement to relational data 11 11
  • 12. Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 12 12
  • 13. Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 13 13
  • 14. Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 14 14
  • 15. Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 15 15
  • 16. Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 16 16
  • 17. Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 17 17
  • 18. Relational vs. document person _id name age department 1 Joe 30 dev 2 Fred 23 dev 3 Jack 26 sales 4 Sue 25 support 5 Pete 32 sales Document Continuent 2013 Tuesday, October 15, 13 18 18
  • 19. How MongoDB keeps data Continuent 2013 Tuesday, October 15, 13 three levels: dbs collections documents 19 19
  • 20. MongoDB insertion demo > show collections > > db.person.insert( {_id: 1, name: "Joe", age: 30, department: "dev"}) > show collections person system.indexes Continuent 2013 Tuesday, October 15, 13 20 20
  • 21. MongoDB insertion demo > db.person.insert( "dev"}) > db.person.insert( "sales"}) > db.person.insert( "support"}) > db.person.insert( "sales"}) > db.person.find() { "_id" : 1, "name" { "_id" : 2, "name" { "_id" : 3, "name" { "_id" : 4, "name" { "_id" : 5, "name" Continuent 2013 Tuesday, October 15, 13 {_id: 2, name: "Fred", age: 23, department: {_id: 3, name: "Jack", age: 26, department: {_id: 4, name: "Sue", age: 25, department: {_id: 5, name: "Pete", age: 30, department: : : : : : "Joe", "age" : 30, "department" : "dev" } "Fred", "age" : 23, "department" : "dev" } "Jack", "age" : 26, "department" : "sales" } "Sue", "age" : 25, "department" : "support" } "Pete", "age" : 30, "department" : "sales" } 21 21
  • 22. MySQL to MongoDB basics Continuent 2013 Tuesday, October 15, 13 22 22
  • 23. Replication from MySQL to MongoDB Continuent 2013 Tuesday, October 15, 13 Requires ROW-based-replication Replication happens by table There is no consolidation into "documents" DDL commands are ignored Statement commands are ignored Column names become document attributes enum and set columns are converted to strings 23 23
  • 24. First example of replication # MySQL create schema oneschema; use oneschema ; create table myfirst( num int not null primary key, dt datetime, ts timestamp, going enum('yes', 'no')); # MongoDB > show dbs local 0.078125GB test 0.203125GB tungsten_mysql2mongodb 0.203125GB # NOTICE: no "oneschema" Continuent 2013 Tuesday, October 15, 13 24 24
  • 25. Inserting data # MySQL insert into myfirst values (1, '2003-04-26 09:15:00', null, 'yes'); Query OK, 1 row affected (0.01 sec) select * from myfirst; +-----+---------------------+---------------------+-------+ | num | dt | ts | going | +-----+---------------------+---------------------+-------+ | 1 | 2003-04-26 09:15:00 | 2013-10-14 19:39:38 | yes | +-----+---------------------+---------------------+-------+ 1 row in set (0.00 sec) Continuent 2013 Tuesday, October 15, 13 25 25
  • 26. Checking results in MongoDB # MongoDB > show dbs local 0.078125GB oneschema 0.203125GB test 0.203125GB tungsten_mysql2mongodb 0.203125GB > use oneschema switched to db oneschema > show collections myfirst system.indexes > db.myfirst.find() { "_id" : ObjectId("525c2c5af5d9ca820fcee01d"), "num" : "1", "dt" : "2003-04-26 11:15:00.0", "ts" : "2013-10-14 19:39:38.0", "going" : "yes" } Continuent 2013 Tuesday, October 15, 13 26 26
  • 27. Another interesting insertion #MySQL create table t1(_id int not null primary key, c char(10)); insert into t1 values (1, 'abc'); select * from t1; +-----+------+ | _id | c | +-----+------+ | 1 | abc | +-----+------+ 1 row in set (0.00 sec) # MongoDB > db.t1.find() { "_id" : "1", "c" : "abc" } Continuent 2013 Tuesday, October 15, 13 27 27
  • 28. More insertions # MySQL insert into t1 values (2,'def'), (3,'ghi'), (4,'jkl'), (5, 'mno'); Query OK, 4 rows affected (0.01 sec) Records: 4 Duplicates: 0 Warnings: 0 select * from t1; +---