Use Your MySQL Knowledge to Become a MongoDB Guru

®

Use Your MySQL Knowledge to Become a MongoDB Guru

Percona Live London 2013

Robert HodgesCEO

Continuent

Tim CallaghanVP/Engineering

Tokutek

Tuesday, November 12, 13

®

Our Companies

Robert Hodges• CEO at Continuent• Database nerd since 1982 starting with M204, RDBMS since

1990, NoSQL since 2012; designed Continuent Tungsten• Continuent offers clustering and replication for MySQL and

other fine DBMS types

Tim Callaghan• VP/Engineering at Tokutek• Long time database consumer (Oracle) and producer (VoltDB,

Tokutek)• Tokutek offers Fractal Tree indexes in MySQL (TokuDB) and

MongoDB (TokuMX)


®

MongoDB -- The New MySQL

One Bad Thing about MongoDB

One Good Thing about MongoDB


®

One Bad Thing about MongoDB

MySQL> select * from table1 where column1 > column2;> ... 5 row(s) returned

MongoDB> db.collection1.find({$field1: {gt: $field2}});> ReferenceError: $field2 is not defined

[current] MongoDB query language is <field> <operator> <literal>


®

One Good Thing about MongoDB

Robert’s “ease of use” demo


®

Today’s Question

How can you use your MySQL knowledge to get up to speed on MongoDB?


®

Topic:

Schema Design


®

How Do I Find Things in MongoDB?

8

mongod server

database

collectionBSON document

key/value pair

key/value pair

key/value pair

BSON document...

== mysqld

== MySQL schema

== MySQL table~ Sort of like a MySQL row

!= MySQL column


®

How Do I Create a Table and Insert Data?

# Ruby CodeMongoClient.new("localhost"). db("mydb"). collection("sample"). insert({"data" => "hello world"})

9

Connect

Primary key generated

automatically

Use database

Choose collection

Insert data to materialize database and collection


®

How Do I Change the Schema?

# Ruby CodeMongoClient.new("localhost"). db("mydb"). collection("sample"). insert({"data" => "hello again!",

"author" => “robert”})

10

Just add more data


®

How Do I Validate Schema?

rs0:PRIMARY> db.samples.find(){ "_id" : 1, "data" : "hello world" }{ "_id" : 2, "daata" : "bye world” }{ "_id" : 3, "data" : 26.44 }

rs0:PRIMARY> show databaseslocal !2.0771484375GBmydb! 7.9501953125GBmydb1! 0.203125GB

11

Typo from an early run

Software bugs?


®

How Do I Remove Data? (Part 1)

12

Drop a databasers0:PRIMARY> db.dropDatabase(){ "dropped" : "mydb", "ok" : 1 }

Drop a collectionrs0:PRIMARY> db.samples.drop()true

Drop a column?rs0:PRIMARY> db.foo.update( { author: { $exists: true }}, { $unset: { author: 1 } }, false, true )


®

How Do I Remove Data? (Part 2)

13

(Remove documents based on TTL index)> db.samples.ensureIndex( {"inserted": 1}, {"expireAfterSeconds": 60})> db.samples.insert( {"data": "hello world", inserted: new Date()})> db.table.count()1...> db.table.count()0

(Capped collections do same with space)


®

How Does MongoDB Do Joins?

14

It Doesn’t!(It is your job to denormalize or do

application level joins. This includes thinking about storage.)


®

Topic:

Data Storageand Organization


®

How is My Data Stored, Logically?

MongoDB storage is very similar to MyISAM

16

_id indexsecondary index(es)

collection data (documents)

etc.


®

How is My Data Stored, Physically?

But it does look different in the file system.

17

• start MongoDB with “--directoryperdb” to put files in database folders• pro-tip : do this to gain IOPs by database

MyISAM<db>/<table>.frm<db>/<table>.myd <db>/<table>.myi

MongoDB<db1>.ns<db1>.1 .. <db1>.n<db2>.ns<db2>.1 .. <db2>.n


®

How Much Memory Does It Use?

All of it!

18


®

How does MongoDB Manage Memory?

• MyISAM–key_cache_size determines index caching–data is cached in Operating System buffers

• InnoDB– innodb_buffer_pool_size determines index/data

caching• MongoDB–memory mapped files–mongod grows to consume available RAM–good : no knob–bad : operating system is in charge of cache–bad : available RAM may change over time

19


®

How Will It Perform for My Workload?

• It depends...–Determine your “working set”

oThe portion of your data that clients access most oftenodb.runCommand( { serverStatus: 1, workingSet: 1 } )

– If working set <= RAMoPerformance generally very goodoBe careful in high-concurrent-write use cases

– If working set >= RAMoLikely IO boundoSharding to the rescue!

20


®

How Can Schema Affect Working Set?

• Field names are stored with the document– On disk and in memory

• Plan ahead, specially for large collections

21

BAD!

{ first_name: “Timothy”, middle_initial: “M”, last_name: “Callaghan”, address_line_1: “555 Main Street”, address_line_2: “Apt. 9” }

GOOD!

{ fn: “Timothy”, mi: “M”, ln: “Callaghan”, al1: “555 Main Street”, al2: “Apt. 9” }


®

Topic:

Query Optimization


®

How Does the Query Optimizer Work?

• MySQL–Optimizer find useable indexes for the query–For each index, optimizer asks the storage engine

oWhat is the cardinality for the given keys?oWhat is the estimated cost?

–The “best” plan is chosen and used for the query

• This occurs for every single query

23


®

How Does the Query Optimizer Work?

• MongoDB–All candidate indexes run the query in parallel

o“candidate” meaning it contains useful keys

–As matching results are found they are placed in a shared buffer–When one of the parallel runs completes, all

others are stopped–This “plan” is used for future executions of the

same queryoUntil the collection has 1,000 writes, mongod restarts, or

there is an index change to the collection

24


®

A Simple Yet Elegant Solution?

• No more wrestling with the optimizer• Hints are supported ($hint)–Force a particular index–http://docs.mongodb.org/manual/reference/

operator/meta/hint/• Easier since MongoDB does not support joins

25


http://docs.mongodb.org/manual/reference/operator/meta/hint/






®

Topic:

Transactions


®

MySQL Transactions and Isolation

mysql> BEGIN;...mysql> INSERT INTO sample(data) VALUES (“Hello world!”);mysql> INSERT INTO sample(data) VALUES (“Goodbye world!”);...mysql> COMMIT;

27

MyISAM locks table and commits each row immediately

InnoDB creates MVCC view of data; locks updated rows, commits atomically


®

How Does MongoDB Implement Locking?

# Update data ranges of documents to # show effects of database lock. @col.update( {key => {"$gte" => first.to_s, "$lt" => last.to_s} }, { "$set" => { "data.x" => rand(@rows)}})

28

Test Total Requests/SecSingle thread updating single collection 197Two threads updating two collections, same DB 80 + 80 = 160Four threads updating two collections, same DB 29+29+30+30 = 118Two threads updating two collections, different DBs 190 + 179 = 369

Locks database


®

How Does MongoDB Implement Isolation?

29

• MongoDB does not prevent threads from seeing partially committed data

• Example: Index changes can result in “double read” of data if query uses index while index is changing

• Experiment: Construct a test to:• Select from numeric index and count rows• Simultaneously update index to shift lower

values past end of previous high value


®

How Does MongoDB Implement Isolation?

# Select values.count = [email protected](“k1” => {"$gte" => 120000}). each do |doc| count += 1endputs "Count=#{count}"

Count=50000Count=50000Count=100000 <--Index shifts over tailCount=50000Count=50000

30

# Run update to [email protected]( {"_id" => {"$exists" => true}}, {"$inc" => {“k1” => increment}}, {:multi => true})


®

Topic:

Replication and HA


®

Review of MySQL Replication

32

Master Slaveset globalread_only=1;

BinlogRelay Log

BinlogRelay Log

Master-master configuration for fast failover


SECOND-ARY

®

How Does MongoDB Set Up Replication?

33

SECOND-ARY

PRIMARY

ReplicationReplication

Heartbeat

Heartbeat


®

Where Is The Replica Set Defined?

$ mongo localhost...# rs0:PRIMARY> rs.config(){! "_id" : "rs0",! "version" : 8,! "members" : [! ! {! ! ! "_id" : 0,! ! ! "host" : "mongodb1:27017"! ! },! ! {! ! ! "_id" : 1,! ! ! "host" : "mongodb2:27017"! ! },! ! {! ! ! "_id" : 2,! ! ! "host" : "mongodb3:27017”! ! }! ]}

34


®

How Do Applications Connect?

# Connect to MongoDB replica set. client = MongoReplicaSetClient.new( ['mongodb1', 'mongodb2', 'mongodb3'])

# Access a collection and add datadb = client.db("xacts")col = db.collection("data")col.insert({"data" => "hello world"})

35


®

How Do You Read From a Slave?

# Connect to MongoDB replica set. client = MongoReplicaSetClient.new( ['mongodb1', 'mongodb2', 'mongodb3'], :slave_ok => true)

# Access a collection and select documents.db = client.db("xacts")col = db.collection("data")col.find()

36


®

Where’s the Binlog?

rs0:PRIMARY> use localrs0:PRIMARY> db.oplog.rs.find().... sort({ts:-1}).limit(1)

{ "ts" : Timestamp(1383980308, 1), "h" : NumberLong("9112507265624716453"), "v" : 2, "op" : "i", "ns" : "xacts.data", "o" : { "_id" : ObjectId("527ddd116244f28f4592f6a8"), "data" : "hello world!" } }

37

Find last document in the OpLog


®

How Do You Lock the DB to Back Up?

38

(= FLUSH TABLES WITH READ LOCK)rs0:SECONDARY> db.fsyncLock(){! "info" : "now locked against writes, use db.fsyncUnlock() to unlock",! "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand",! "ok" : 1}...(tar or rsync data)...(= UNLOCK TABLES)rs0:SECONDARY> db.fsyncUnlock(){ "ok" : 1, "info" : "unlock completed" }


http://dochub.mongodb.org/core/fsynccommand




®

How Do You Fail Over?

39

• Planned failover: update rs.config and save:rs0:SECONDARY> cfg = rs.conf()

rs0:SECONDARY> cfg.members[0].priority = 1rs0:SECONDARY> cfg.members[1].priority = 1

rs0:SECONDARY> cfg.members[2].priority = 2

rs0:SECONDARY> rs.reconfig(cfg)

• Unplanned failover: kill or stop mongod


®

Topic:

Sharding


®

How is Partitioning Like Sharding?

• MySQL partitioning breaks a table into <n> tables– “PARTITION” is actually a storage engine

• Tables can be partitioned by hash or range–Hash = random distribution–Range = user controlled distribution (date range)

• Helpful in “big data” use-cases• Partitions can usually be dropped efficiently–Unlike “delete from table1 where timeField < ’12/31/2012’;”

41


®

How Does Partitioning Help Queries?

select * from big_table where column1 = 5;

42

select * from big_table where dateCol = ’10/12/2013’;

Aug-2013 Sep-2013 Oct-2013 Nov-2013

Partitioned big_table on dateCol by month.


®

Can I Finally Scale My Workload Horizontally?

• MySQL partitioning is helpful, but is still constrained to a single machine

• MongoDB supports cross-server sharding–huge plus: it’s “in the box”–MySQL fabric is bringing something, we’ll see–Many other 3rd Party MySQL options exist

• Only shard the collections that require it• Each MongoDB shard is a replica set (1

primary and 1+ secondaries)

43


®

What Does MongoDB Sharding Look Like?

44

Master Slave shard1

Master Slave shard2

Master Slave shardn

...

mongos1

mongosn

...

client app1

client app2

...


®

How Does Sharding Help Queries?

select * from big_table where column1 = 5;

45

select * from big_table where dateCol = ’10/12/2013’;

Aug-2013 Sep-2013 Oct-2013 Nov-2013

Sharded big_table on dateCol by month.

shard1 shard2 shard3 shard4 shard...shard...


®

How Do I Pick a Shard Key?

• MongoDB shards on one or more fields• Simple example– “orders” collection (customerId and productId)–1: shard on customerId

oeach order writes to a single shardo reads by customer on single shardo reads by product on entire cluster

–2: shard on productIdoeach order writes to several shardso reads by customer on entire clustero reads by product on single shard

–3: store everything twice and shard both waysoworst case for writesobest cast for reads (either is shingle shard)

46


®

Topic:

Security


®

How Secure is It?

• basic username/password• by database• roles– read = read any collection– readWrite = read/write any collection–dbAdmin = create index, create collection, rename

collection, etc.

48


®

What About Advanced Security?

• Kerberos support in MongoDB Enterprise Edition

• SSL is supported, but–Note: The default distribution of MongoDB does

not contain support for SSL. To use SSL, you must either build MongoDB locally passing the “--ssl” option to scons or use MongoDB Enterprise.

49


®

What Else is There to Learn?

• Tools - mongostat, mongo[export/import], mongo[dump/restore]• Aggregation Framework• Think SQL aggregate functionality• Map/Reduce


®

What Should You Do?


®

Summary

We liked...• Ease of install• Ability to just “jump in”

Look [out] for...• Query language (Tim says hang in there!)• You have to think about storage and queries in advance

Highly Recommended Reading• Karl Seguin’s “The Little MongoDB Book”

• http://openmymind.net/mongodb.pdf

• MongoDB’s “SQL to MongoDB Mapping Chart”• http://docs.mongodb.org/manual/reference/sql-comparison/


http://openmymind.net/mongodb.pdf

http://openmymind.net/mongodb.pdf

http://docs.mongodb.org/manual/reference/sql-comparison/

http://docs.mongodb.org/manual/reference/sql-comparison/

®

Questions?

Tim CallaghanVP/Engineering, Tokutek

[email protected]@tmcallaghan

Robert HodgesCEO, Continuent

[email protected]@continuent


mailto:[email protected]




Use Your MySQL Knowledge to Become a MongoDB Guru

Technology

mongodb mysql

mongodb db

mongodb storage

data storage

mongodb tokumx

collectioninsert data

mysql schema

mysql table