Top Banner
Migrating data to MongoDB
21

Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

Nov 19, 2014

Download

Technology

MongoDB

Now that you've chosen MongoDB, how should you migrate your deployment? In this webinar, we'll discuss the different ways companies migrate to MongoDB including hybrid solutions, where only new or certain features are migrated, and all-in migrations, where everything is migrated. We'll use real customer examples to discuss the pros and cons of each so you can decide what strategy will work best for your deployment.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

Migrating data to MongoDB

Page 2: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

2

Agenda

• Why move your data

• Considerations for migration

• Techniques for implementing migration

• Case study: How Shutterfly migrated 20TB of production meta data with no downtime

Page 3: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

3

Why move your data?

• Improve development agility with documents

• Reduce cost of data management

• Scale to handle large data sets & transaction volumes

Page 4: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

Considerations

Page 5: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

5

Does your schema need to change?

Relational MongoDB{ first_name: ‘Paul’, surname: ‘Miller’ city: ‘London’, location: [45.123,47.232], cars: [ { model: ‘Bentley’, year: 1973, value: 100000, … }, { model: ‘Rolls Royce’, year: 1965, value: 330000, … } ]}

Page 6: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

6

How does the Application move over?

Source Database

Application

Page 7: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

7

How does data get moved?

Source Database

Snapshot

Continuous Sync

Batch Migration

Application

Application Managed

Page 8: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

8

Can you have downtime?

Source Database Master Exporting

Importing Master

Available Degraded Down AvailableApplication View

Time

Page 9: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

Data Migration Technique

Page 10: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

10

Mongoimport

jsr@bruford:/tmp$ mongoimport --collection import_example < import.json connected to: 127.0.0.1Tue Jun 18 00:02:12.553 imported 1 objectsjsr@bruford:/tmp$ mongo MongoDB shell version: 2.4.3connecting to: test> db.import_example.findOne() {

"_id" : ObjectId("51bfdbc438b61619a4f2a12b"),"first" : "Jared","last" : "Rosoff","twitter" : "@forjared"

}>

Page 11: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

11

Extract, Transform & Load

Source DatabaseETL

Page 12: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

12

Hadoop

Source Database

jobjob

jobjob

Page 13: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

13

App Driven Migration

Source Database

Application

Page 14: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

Case Study

Page 15: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

15

Case Study

Uses MongoDB to safeguard over 6 billion images served to millions of customers

Problem Why MongoDB Results

• 6B images, 20TB of data

• Brittle code base on top of Oracle database – hard to scale, add features

• High SW and HW costs

• JSON-based data model

• Agile, high performance, scalable

• Alignment with Shutterfly’s services-based architecture

• 80% cost reduction

• 900% performance improvement

• Faster time-to-market

• Dev. cycles in weeks vs. tens of months

Page 16: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

16

• Meta data stored in XML Blobs

• App responsible for content of blob

Shutterfly

Oracle

Photo ID

XML Blob

1 <xml><meta-data>…</xml>

2 <xml><meta-data>…</xml>

3 <xml><meta-data>…</xml>

Page 17: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

17

1. Request for photo

2. Try to read from MongoDB

3. If cache miss, read from Oracle

4. Translate document & write to MongoDB

5. Return to client

Migrating records on demand

Source Database

Application

1

23

4

5

Page 18: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

18

Schema Migration – Initial

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

Page 19: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

19

Schema Migration – Phase 2

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }

Page 20: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB

20

Schema Migration – Phase 2

<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }

{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "votes" : { 000000000:1, 111111111:1, 222222222:1 }}

Page 21: Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB