Optimising for performance, scale, analytics
MongoDB: Optimising for Performance, Scale & Analytics
Jun 27, 2015
MongoDB is easy to download and run locally but requires some thought and further understanding when deploying to production. At scale, schema design, indexes and query patterns really matter. So does data structure on disk, sharding, replication and data centre awareness. This talk will examine these factors in the context of analytics, and more generally, to help you optimise MongoDB for any scale.
Presented at MongoDB Days London 2013 by David Mytton.
Presented at MongoDB Days London 2013 by David Mytton.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Optimising for performance, scale, analytics
~3333 write ops/s 0.07 - 0.05 ms response
David Mytton
Woop Japan!
MongoDB at Server Density
•27 nodes
MongoDB at Server Density
• June 2009 - +3yrs
•27 nodes
MongoDB at Server Density
•MySQL -> MongoDB
•27 nodes
MongoDB at Server Density
• June 2009 - +3yrs
•MySQL -> MongoDB
•17TB data per month
•27 nodes
MongoDB at Server Density
• June 2009 - +3yrs
MongoDB at Server Density
Queues
Primary data store
Time series
Why?
Why?
• Replication
Why?
• Replication
• Official drivers
Why?
• Replication
• Official drivers
• Easy deployment
Why?
• Replication
• Official drivers
• Easy deployment
• Fast out of the box (sort of)
Fast out of the box?
Photo: dannychoo.com
~3333 write ops/s 0.07 - 0.05 ms response
Fast out of the box?
Photo: dannychoo.com
• Softlayer cloud (1 core, 8GB)
Fast out of the box?
Photo: dannychoo.com
• Local instance storage
• Softlayer cloud (1 core, 8GB)
Fast out of the box?
Photo: dannychoo.com
• Local instance storage
• Ubuntu 10/12.04 LTS
• Softlayer cloud (1 core, 8GB)
Fast out of the box?
Photo: dannychoo.com
• Journaling
Fast out of the box?
Photo: dannychoo.com
• Journaling
• Replication
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
• Fast network
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
• Fast network
• Working set in RAM
mongos> db.metrics_20120508_15_1m.stats(){ "sharded" : true, "flags" : 1, "ns" : "metrics.metrics_20120508_15_1m", "count" : 2752934, "numExtents" : 46,
"size" : 746837640, "storageSize" : 823717888,
"totalIndexSize" : 517581680, "indexSizes" : { "_id_" : 130358144, "a_1_i_1" : 155711920, "a_1_i_1_m_1_t_1" : 231511616 }, "avgObjSize" : 271.2878841265355, "nindexes" : 3, "nchunks" : 61,
"size" : 746837640, "totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB
"size" : 746837640, "totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB
"size" : 746837640, "totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB
Total = 1205MB
http://www.flickr.com/photos/comedynose/4388430444/
Where should it go?
What? Should it be in memory?
Indexes Always
Data If you can
How you’ll know
1) Slow queries
Thu Oct 14 17:01:11 [conn7410] update sd.apiLog query: { c: "android/setDeviceToken", a: 1466, u: "blah", ua: "Server Density Android" } 51926ms
www.flickr.com/photos/tonivc/2283676770/
How you’ll know
2) Timeouts
cursor timed out (20000 ms)
How you’ll know
3) Disk i/o spikes
www.flickr.com/photos/daddo83/3406962115/
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
• Fast network
• Working set in RAM
• Fast disks (optional)
Growing documents is bad
Updates in place
db.my_collection.update( { _id : ... }, { $inc : { y : 2 } } )
BSON
BSON
Growing documents is bad
> db.coll.stats(){ "ns" : "...", ..., "paddingFactor" : 1, ..., "ok" : 1}
• Global DB lock
Scaling writes
• Concurrency
Scaling writes
• Global DB lock
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
• Fast network
• Working set in RAM
• Fast disks (optional)
• Sharding (optional)
• Sharding
• Concurrency
Scaling writes
• Global DB lock
Scaling writes
• Collection location
Scaling writes
• Collection location
Scaling writes
• Pre-split / moveChunk
• Collection location
Scaling writes
• Hashing (v2.4)
• Pre-split / moveChunk
Failover
•Replica sets
Failover
•Replica sets
•Master/slave
Failover
•Replica sets
•Min 3 nodes
•Master/slave
Failover
•Replica sets
•Min 3 nodes
•Master/slave
•Automatic failover
rs.status()
www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek)
{! "_id" : 1,! "name" : "rs3b:27018",! "health" : 1,! "state" : 2,! "stateStr" : "SECONDARY",! "uptime" : 1886098,! "optime" : {! ! "t" : 1291252178000,! ! "i" : 13! },! "optimeDate" : ISODate("2010-12-02T01:09:38Z"), "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")},
1) myState
rs.status()
Value Meaning0 Starting up (phase 1)1 Primary2 Secondary3 Recovering4 Fatal error5 Starting up (phase 2)6 Unknown state7 Arbiter8 Down
2) Optime
rs.status()
www.flickr.com/photos/robbie73/4244846566/
"optimeDate" : ISODate("2010-12-02T01:09:38Z")
3) Heartbeat
rs.status()
www.flickr.com/photos/drawblindfaith/3400981091/
"lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")
Scaling reads
Scaling reads
•Replica slaves
Scaling reads
•Replica slaves
•Consistency
•Replica slaves
Scaling reads
•Consistency
•w flag / tags
WriteConcern
Changed Nov 27 2012
• Safe by default
WriteConcern
>>> from pymongo import MongoClient>>> connection = MongoClient()
• Safe by default
WriteConcern
>>> from pymongo import MongoClient>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Primary
2 Primary + x1 secondary
3 Primary + x2 secondaries
Tags
{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
(A or B) + (C or D) + E
Tags
{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "sortOfImportant"})
(A + C) or (D + E) ...
Tags
• J
WriteConcern
• Safe by default
• J
• fsync
WriteConcern
• Safe by default
Bottlenecks
•EC2
Bottlenecks
•EC2
•Local storage
Bottlenecks
•EC2
•EBS: RAID10 4-8 volumes
•Local storage
Bottlenecks
•EC2
•EBS: RAID10 4-8 volumes
•Local storage
•i/o: rand but not sequential
http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
http://bit.ly/ec2mongodb
Bottlenecks
•CPU
•Index building
Tips: rand()
•_id
Tips: rand()
•_id
•Field names
Tips: rand()
•_id
•Field names
•Covered indexes
Tips: rand()
•_id
•Field names
•Covered indexes
•Collections / databases
mongostat
Locks/Queues
mongostat
Diagnostics
mongostat
Current operations
www.flickr.com/photos/jeffhester/2784666811/
db.currentOp();{! ! ! "opid" : "shard1:299939199",! ! ! "active" : true,! ! ! "lockType" : "write",! ! ! "waitingForLock" : false,! ! ! "secs_running" : 15419,! ! ! "op" : "remove",! ! ! "ns" : "sd.metrics",! ! ! "query" : {! ! ! ! "accId" : 1391,! ! ! ! "tA" : {! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z")! ! ! ! }! ! ! },! ! ! "client" : "10.121.12.228:44426",! ! ! "desc" : "conn"! ! },
Monitoring tools
Run yourself
Ganglia
Monitoring tools
Server Density
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.
Fast out of the box?
• Fast network
• Working set in RAM
• bit.ly/benchrun
Woop Japan!
www.serverdensity.com/mdb
Related Documents
