MongoDB: Optimising for Performance, Scale & Analytics

Optimising for performance, scale, analytics

~3333 write ops/s 0.07 - 0.05 ms response

David Mytton

Woop Japan!

MongoDB at Server Density

•27 nodes

MongoDB at Server Density

• June 2009 - +3yrs

•27 nodes

MongoDB at Server Density

•MySQL -> MongoDB

•27 nodes

MongoDB at Server Density

• June 2009 - +3yrs

•MySQL -> MongoDB

•17TB data per month

•27 nodes

MongoDB at Server Density

• June 2009 - +3yrs

MongoDB at Server Density

Queues

Primary data store

Time series

Why?

Why?

• Replication

Why?

• Replication

• Official drivers

Why?

• Replication

• Official drivers

• Easy deployment

Why?

• Replication

• Official drivers

• Easy deployment

• Fast out of the box (sort of)

Fast out of the box?

Photo: dannychoo.com

~3333 write ops/s 0.07 - 0.05 ms response

Fast out of the box?

Photo: dannychoo.com

• Softlayer cloud (1 core, 8GB)

Fast out of the box?

Photo: dannychoo.com

• Local instance storage

• Softlayer cloud (1 core, 8GB)

Fast out of the box?

Photo: dannychoo.com

• Local instance storage

• Ubuntu 10/12.04 LTS

• Softlayer cloud (1 core, 8GB)

Fast out of the box?

Photo: dannychoo.com

• Journaling

Fast out of the box?

Photo: dannychoo.com

• Journaling

• Replication

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

• Fast network

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

• Fast network

• Working set in RAM

mongos> db.metrics_20120508_15_1m.stats(){ "sharded" : true, "flags" : 1, "ns" : "metrics.metrics_20120508_15_1m", "count" : 2752934, "numExtents" : 46,

"size" : 746837640, "storageSize" : 823717888,

"totalIndexSize" : 517581680, "indexSizes" : { "_id_" : 130358144, "a_1_i_1" : 155711920, "a_1_i_1_m_1_t_1" : 231511616 }, "avgObjSize" : 271.2878841265355, "nindexes" : 3, "nchunks" : 61,

"size" : 746837640, "totalIndexSize" : 517581680

Indexes = 493MB Data = 712MB

"size" : 746837640, "totalIndexSize" : 517581680

Indexes = 493MB Data = 712MB

"size" : 746837640, "totalIndexSize" : 517581680

Indexes = 493MB Data = 712MB

Total = 1205MB

http://www.flickr.com/photos/comedynose/4388430444/

Where should it go?

What? Should it be in memory?

Indexes Always

Data If you can

How you’ll know

1) Slow queries

Thu Oct 14 17:01:11 [conn7410] update sd.apiLog query: { c: "android/setDeviceToken", a: 1466, u: "blah", ua: "Server Density Android" } 51926ms

www.flickr.com/photos/tonivc/2283676770/

How you’ll know

2) Timeouts

cursor timed out (20000 ms)

How you’ll know

3) Disk i/o spikes

www.flickr.com/photos/daddo83/3406962115/

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

• Fast network

• Working set in RAM

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

• Fast network

• Working set in RAM

• Fast disks (optional)

Growing documents is bad

Updates in place

db.my_collection.update( { _id : ... }, { $inc : { y : 2 } } )

BSON

BSON

Growing documents is bad

> db.coll.stats(){ "ns" : "...", ..., "paddingFactor" : 1, ..., "ok" : 1}

• Global DB lock

Scaling writes

• Concurrency

Scaling writes

• Global DB lock

http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

http://bit.ly/mongolock

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

• Fast network

• Working set in RAM

• Fast disks (optional)

• Sharding (optional)

• Sharding

• Concurrency

Scaling writes

• Global DB lock

Scaling writes

• Collection location

Scaling writes

• Collection location

Scaling writes

• Pre-split / moveChunk

• Collection location

Scaling writes

• Hashing (v2.4)

• Pre-split / moveChunk

Failover

•Replica sets

Failover

•Replica sets

•Master/slave

Failover

•Replica sets

•Min 3 nodes

•Master/slave

Failover

•Replica sets

•Min 3 nodes

•Master/slave

•Automatic failover

rs.status()

www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek)

{! "_id" : 1,! "name" : "rs3b:27018",! "health" : 1,! "state" : 2,! "stateStr" : "SECONDARY",! "uptime" : 1886098,! "optime" : {! ! "t" : 1291252178000,! ! "i" : 13! },! "optimeDate" : ISODate("2010-12-02T01:09:38Z"), "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")},

1) myState

rs.status()

Value Meaning0 Starting up (phase 1)1 Primary2 Secondary3 Recovering4 Fatal error5 Starting up (phase 2)6 Unknown state7 Arbiter8 Down

2) Optime

rs.status()

www.flickr.com/photos/robbie73/4244846566/

"optimeDate" : ISODate("2010-12-02T01:09:38Z")

3) Heartbeat

rs.status()

www.flickr.com/photos/drawblindfaith/3400981091/

"lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")

Scaling reads

Scaling reads

•Replica slaves

Scaling reads

•Replica slaves

•Consistency

•Replica slaves

Scaling reads

•Consistency

•w flag / tags

WriteConcern

Changed Nov 27 2012

• Safe by default

WriteConcern

>>> from pymongo import MongoClient>>> connection = MongoClient()

• Safe by default

WriteConcern

>>> from pymongo import MongoClient>>> connection = MongoClient(w=int/str)

Value Meaning

0 Unsafe

1 Primary

2 Primary + x1 secondary

3 Primary + x2 secondaries

Tags

{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})

{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})

(A or B) + (C or D) + E

Tags

{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "sortOfImportant"})

(A + C) or (D + E) ...

Tags

• J

WriteConcern

• Safe by default

• J

• fsync

WriteConcern

• Safe by default

Bottlenecks

•EC2

Bottlenecks

•EC2

•Local storage

Bottlenecks

•EC2

•EBS: RAID10 4-8 volumes

•Local storage

Bottlenecks

•EC2

•EBS: RAID10 4-8 volumes

•Local storage

•i/o: rand but not sequential

http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

http://bit.ly/ec2mongodb

Bottlenecks

•CPU

•Index building

Tips: rand()

•_id

Tips: rand()

•_id

•Field names

Tips: rand()

•_id

•Field names

•Covered indexes

Tips: rand()

•_id

•Field names

•Covered indexes

•Collections / databases

mongostat

Locks/Queues

mongostat

Diagnostics

mongostat

Current operations

www.flickr.com/photos/jeffhester/2784666811/

db.currentOp();{! ! ! "opid" : "shard1:299939199",! ! ! "active" : true,! ! ! "lockType" : "write",! ! ! "waitingForLock" : false,! ! ! "secs_running" : 15419,! ! ! "op" : "remove",! ! ! "ns" : "sd.metrics",! ! ! "query" : {! ! ! ! "accId" : 1391,! ! ! ! "tA" : {! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z")! ! ! ! }! ! ! },! ! ! "client" : "10.121.12.228:44426",! ! ! "desc" : "conn"! ! },

Monitoring tools

Run yourself

Ganglia

Monitoring tools

Server Density

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

• Fast network

• Working set in RAM

Picture is unrelated! Mmm, ice cream.

Fast out of the box?

• Fast network

• Working set in RAM

• bit.ly/benchrun

Woop Japan!

www.serverdensity.com/mdb

David Mytton

david@serverdensity.com

@davidmytton

Woop Japan!

www.serverdensity.com