Mongo db

MongoDBhttp://tinyurl.com/97o49y3

by toki

About me

● Delta Electronic CTBD Senior Engineer● Main developer of http://loltw.net

○ Website built via MongoDB with daily 600k PV○ Data grow up everyday with auto crawler bots

MongoDB - Simple Introduction

● Document based NOSQL(Not Only SQL) database

● Started from 2007 by 10Gen company● Wrote in C++● Fast (But takes lots of memory)● Stores JSON documents in BSON format● Full index on any document attribute● Horizontal scalability with auto sharding● High availability & replica ready

What is database?

● Raw data○ John is a student, he's 12 years old.

● Data○ Student

■ name = "John"■ age = 12

● Records○ Student(name="John", age=12)○ Student(name="Alice", age=11)

● Database○ Student Table○ Grades Table

Example of (relational) database

Student

Student ID

Name

Age

Class ID

Class

Class ID

Name

Student Grade

Grade ID

StudentID

GradeGrade

Grade ID

Name

SQL Language - How to find data?

● Find student name is John○ select * from student where name="John"

● Find class name of John○ select s.name, c.name as class_name from student

s, class c where name="John" and s.class_id=c.class_id

Why NOSQL?

● Big data○ Morden data size is too big for single DB server○ Google search engine

● Connectivity○ Facebook like button

● Semi-structure data○ Car equipments database

● High availability○ The basic of cloud service

Common NOSQL DB characteristic

● Schemaless● No join, stores pre-joined/embedded data● Horizontal scalability ● Replica ready - High availability

Common types of NOSQL DB

● Key-Value○ Based on Amazon's Dynamo paper○ Stores K-V pairs○ Example:

■ Dynomite■ Voldemort

Common types of NOSQL DB

● Bigtable clones○ Based on Google Bigtable paper○ Column oriented, but handles semi-structured data○ Data keyed by: row, column, time, index○ Example:

■ Google Big Table■ HBase■ Cassandra(FB)

Common types of NOSQL DB

● Document base○ Stores multi-level K-V pairs○ Usually use JSON as document format○ Example:

■ MongoDB■ CounchDB (Apache)■ Redis

Common types of NOSQL DB

● Graph○ Focus on modeling the structure of data -

interconnectivity○ Example

■ Neo4j■ AllegroGraph

Start using MongoDB - Installation

● From apt-get (debian / ubuntu only)○ sudo apt-get install mongodb

● Using 10-gen mongodb repository○ http://docs.mongodb.org/manual/tutorial/install-

mongodb-on-debian-or-ubuntu-linux/● From pre-built binary or source

○ http://www.mongodb.org/downloads● Note:

32-bit builds limited to around 2GB of data

Manual start your MongoDB

mkdir -p /tmp/mongomongod --dbpath /tmp/mongo

or

mongod -f mongodb.conf

Verify your MongoDB installation

$ mongo

MongoDB shell version: 2.2.0connecting to: test>_

--------------------------------------------------------mongo localhost/test2mongo 127.0.0.1/test

How many database do you have?

show dbs

Elements of MongoDB

● Database○ Collection

■ Document

What is JSON

● JavaScript Object Notation● Elements of JSON

○ Object: K/V pairs○ Key, String○ Value, could be

■ string■ bool■ number■ array■ object■ null

{"key1": "value1","key2": 2.0"key3": [1, "str", 3.0],"key4": false,"key5": { "name": "another object",}

}

Another sample of JSON

{"name": "John","age": 12,"grades": {

"math": 4.0,"english": 5.0

},"registered": true,"favorite subjects": ["math", "english"]

}

Insert document into MongoDB

s = {"name": "John","age": 12,"grades": {

"math": 4.0,"english": 5.0

},"registered": true,"favorite subjects": ["math", "english"]

}

db.students.insert(s);

Verify inserted document

db.students.find()

also try

db.student.insert(s)show collections

Save document into MongoDB

s.name = "Alice"s.age = 14s.grades.math = 2.0

db.students.save(s)

What is _id / ObjectId ?

● _id is the default primary key for indexing documents, could be any JSON acceptable value.

● By default, MongoDB will auto generate a ObjectId as _id

● ObjectId is 12 bytes value of unique document _id

● Use ObjectId().getTimestamp() to restore the timestamp in ObjectId

0 1 2 3 4 5 6 7 8 9 10 11

unix timestamp machine process id Increment

Save document with id into MongoDB

s.name = "Bob"s.age = 11s['favorite subjects'] = ["music", "math", "art"]s.grades.chinese = 3.0s._id = 1

db.students.save(s)

Save document with existing _id

delete s.registered

db.students.save(s)

How to find documents?

● db.xxxx.find()○ list all documents in collection

● db.xxxx.find(find spec, //how document looks likefind fields, //which parts I wanna see...

)● db.xxxx.findOne()

○ only returns first document match find spec.

find by id

db.students.find({_id: 1})db.students.find({_id: ObjectId('xxx....')})

find and filter return fields

db.students.find({_id: 1}, {_id: 1})db.students.find({_id: 1}, {name: 1})db.students.find({_id: 1}, {_id: 1, name: 1})db.students.find({_id: 1}, {_id: 0, name: 1})

find by name - equal or not equal

db.students.find({name: "John"})db.students.find({name: "Alice"})

db.students.find({name: {$ne: "John"}})● $ne : not equal

find by name - ignorecase ($regex)

db.students.find({name: "john"}) => Xdb.students.find({name: /john/i}) => O

db.students.find({name: {

$regex: "^b", $options: "i"

}})

find by range of names - $in, $nin

db.students.find({name: {$in: ["John", "Bob"]}})db.students.find({name: {$nin: ["John", "Bob"]}})

● $in : in range (array of items)● $nin : not in range

find by age - $gt, $gte, $lt, $lte

db.students.find({age: {$gt: 12}})db.students.find({age: {$gte: 12}})db.students.find({age: {$lt: 12}})db.students.find({age: {$lte: 12}})

● $gt : greater than● $gte : greater than or equal● $lt : lesser than● $lte : lesser or equal

find by field existence - $exists

db.students.find({registered: {$exists: true}})db.students.find({registered: {$exists: false}})

find by field type - $type

db.students.find({_id: {$type: 7}})db.students.find({_id: {$type: 1}})

1 Double 11 Regular expression

2 String 13 JavaScript code

3 Object 14 Symbol

4 Array 15 JavaScript code with scope

5 Binary Data 16 32 bit integer

7 Object id 17 Timestamp

8 Boolean 18 64 bit integer

9 Date 255 Min key

10 Null 127 Max key

find in multi-level fields

db.students.find({"grades.math": {$gt: 2.0}})db.students.find({"grades.math": {$gte: 2.0}})

find by remainder - $mod

db.students.find({age: {$mod: [10, 2]}})db.students.find({age: {$mod: [10, 3]}})

find in array - $size

db.students.find({'favorite subjects': {$size: 2}}

)db.students.find(

{'favorite subjects': {$size: 3}})

find in array - $all

db.students.find({'favorite subjects': {$all: ["music", "math", "art"]

}})db.students.find({'favorite subjects': {

$all: ["english", "math"]}})

find in array - find value in array

db.students.find({"favorite subjects": "art"}

)

db.students.find({"favorite subjects": "math"}

)

find with bool operators - $and, $or

db.students.find({$or: [{age: {$lt: 12}},{age: {$gt: 12}}

]})

db.students.find({$and: [{age: {$lt: 12}},{age: {$gte: 11}}

]})

find with bool operators - $and, $or

db.students.find({$and: [{age: {$lt: 12}},{age: {$gte: 11}}

]})

equals to

db.student.find({age: {$lt:12, $gte: 11}}

find with bool operators - $not

$not could only be used with other find filter

X db.students.find({registered: {$not: false}})O db.students.find({registered: {$ne: false}})

O db.students.find({age: {$not: {$gte: 12}}})

find with JavaScript- $where

db.students.find({$where: "this.age > 12"})

db.students.find({$where:"this.grades.chinese"

})

find cursor functions

● countdb.students.find().count()

● limitdb.students.find().limit(1)

● skipdb.students.find().skip(1)

● sortdb.students.find().sort({age: -1})db.students.find().sort({age: 1})

combine find cursor functions

db.students.find().skip(1).limit(1)db.students.find().skip(1).sort({age: -1})db.students.find().skip(1).limit(1).sort({age: -1})

more cursor functions

● snapshotensure cursor returns○ no duplicates○ misses no object○ returns all matching objects that were present at

the beginning and the end of the query.○ usually for export/dump usage

more cursor functions

● batchSizetell MongoDB how many documents should be sent to client at once

● explainfor performance profiling

● hinttell MongoDB which index should be used for querying/sorting

list current running operations

● list operationsdb.currentOP()

● cancel operationsdb.killOP()

MongoDB index - when to use index?

● while doing complicate find● while sorting lots of data

MongoDB index - sort() example

for (i=0; i<1000000; i++){ db.many.save({value: i});

}

db.many.find().sort({value: -1})

error: {"$err" : "too much data for sort() with no index. add an index or specify

a smaller limit","code" : 10128

}

MongoDB index - how to build index

db.many.ensureIndex({value: 1})

● Index options○ background○ unique○ dropDups○ sparse

MongoDB index - index commands

● list indexdb.many.getIndexes()

● drop indexdb.many.dropIndex({value: 1})db.many.dropIndexes() <-- DANGER!

MongoDB Index - find() example

db.many.dropIndex({value: 1})db.many.find({value: 5555}).explain()

db.many.ensureIndex({value: 1})db.many.find({value: 5555}).explain()

MongoDB Index - Compound Index

db.xxx.ensureIndex({a:1, b:-1, c:1})

query/sort with fields● a● a, b● a, b, c

will be accelerated by this index

Remove/Drop data from MongoDB

● Removedb.many.remove({value: 5555})db.many.find({value: 5555})db.many.remove()

● Dropdb.many.drop()

● Drop databasedb.dropDatabase() EXTREMELY DANGER!!!

How to update data in MongoDB

Easiest way:

s = db.students.findOne({_id: 1})s.registered = truedb.students.save(s)

In place update - update()

update({find spec},{update spec},upsert=false)

db.students.update({_id: 1},{$set: {registered: false}}

)

Update a non-exist document

db.students.update({_id: 2}, {name: 'Mary', age: 9},true

)db.students.update(

{_id: 2}, {$set: {name: 'Mary', age: 9}},true

)

set / unset field value

db.students.update({_id: 1},{$set: {"age": 15}})

db.students.update({_id: 1},{$set: {registered:

{2012: false, 2011:true}}})

db.students.update({_id: 1},{$unset: {registered: 1}})

increase/decrease value

db.students.update({_id: 1}, {$inc: {

"grades.math": 1.1,"grades.english": -1.5,"grades.history": 3.0

}})

push value(s) into array

db.students.update({_id: 1},{$push: {tags: "lazy"}

})

db.students.update({_id: 1},{$pushAll: {tags: ["smart", "cute"]}

})

add only not exists value to array

db.students.update({_id: 1},{$push: {tags: "lazy"}

})db.students.update({_id: 1},{

$addToSet:{tags: "lazy"}})db.students.update({_id: 1},{

$addToSet:{tags: {$each: ["tall", "thin"]}}})

remove value from array

db.students.update({_id: 1},{$pull: {tags: "lazy"}

})db.students.update({_id: 1},{

$pull: {tags: {$ne: "smart"}}})db.students.update({_id: 1},{

$pullAll: {tags: ["lazy", "smart"]}})

pop value from array

a = []; for(i=0;i<20;i++){a.push(i);}db.test.save({_id:1, value: a})

db.test.update({_id: 1}, {$pop: {value: 1}

})db.test.update({_id: 1}, {

$pop: {value: -1}})

rename field

db.test.update({_id: 1}, {$rename: {value: "values"}

})

Practice: add comments to student

Add a field into students ({_id: 1}):● field name: comments● field type: array of dictionary● field content:

○ {

by: author name, stringtext: content of comment, string

}● add at least 3 comments to this field

Example answer to practice

db.students.update({_id: 1}, {$addToSet: { comments: {$each: [

{by: "teacher01", text: "text 01"},{by: "teacher02", text: "text 02"},{by: "teacher03", text: "text 03"},

]}}})

The $ position operator (for array)

db.students.update({_id: 1,"comments.by": "teacher02"

}, {$inc: {"comments.$.vote": 1}

})

Atomically update - findAndModify

● Atomically update SINGLE DOCUMENT and return it

● By default, returned document won't contain the modification made in findAndModify command.

findAndModify parameters

db.xxx.findAndModify({query: filter to querysort: how to sort and select 1st document in query resultsremove: set true if you want to remove itupdate: update contentnew: set true if you want to get the modified objectfields: which fields to fetchupsert: create object if not exists})

GridFS

● MongoDB has 32MB document size limit● For storing large binary objects in MongoDB● GridFS is kind of spec, not implementation● Implementation is done by MongoDB drivers● Current supported drivers:

○ PHP○ Java○ Python○ Ruby○ Perl

GridFS - command line tools

● Listmongofiles list

● Putmongofiles put xxx.txt

● Getmongofiles get xxx.txt

MongoDB config - basic

● dbpath○ Which folder to put MongoDB database files○ MongoDB must have write permission to this folder

● logpath, logappend○ logpath = log filename○ MongoDB must have write permission to log file

● bind_ip○ IP(s) MongoDB will bind with, by default is all○ User comma to separate more than 1 IP

● port○ Port number MongoDB will use○ Default port = 27017

Small tip - rotate MongoDB log

db.getMongo().getDB("admin").runCommand("logRotate")

MongoDB config - journal

● journal○ Set journal on/off○ Usually you should keep this on

MongoDB config - http interface

● nohttpinterface○ Default listen on http://localhost:28017○ Shows statistic info with http interface

● rest○ Used with httpinterface option enabled only○ Example:

http://localhost:28017/test/students/http://localhost:28017/test/students/?filter_name=John

MongoDB config - authentication

● auth○ By default, MongoDB runs with no authentication○ If no admin account is created, you could login with

no authentication through local mongo shell and start managing user accounts.

MongoDB account management

● Add admin user> mongo localhost/admindb.addUser("testadmin", "1234")

● Authenticated as admin useruse admindb.auth("testadmin", "1234")

MongoDB account management

● Add user to test databaseuse testdb.addUser("testrw", "1234")

● Add read only user to test databasedb.addUser("testro", "1234", true)

● List usersdb.system.users.find()

● Remove user db.removeUser("testro")

MongoDB config - authentication

● keyFile○ At least 6 characters and size smaller than 1KB○ Used only for replica/sharding servers○ Every replica/sharding server should use the same

key file for communication○ On U*ix system, file permission to key file for

group/everyone must be none, or MongoDB will refuse to start

MongoDB configuration - Replica Set

● replSet○ Indicate the replica set name○ All MongoDB in same replica set should use the

same name○ Limitation

■ Maximum 12 nodes in a single replica set■ Maximum 7 nodes can vote

○ MongoDB replica set is Eventually consistent

How's MongoDB replica set working?

● Each a replica set has single primary(master) node and multiple slave nodes

● Data will only be wrote to primary node then will be synced to other slave nodes.

● Use getLastError() for confirming previous write operation is committed to whole replica set, otherwise the write operation may be rolled back if primary node is down before sync.

How's MongoDB replica set working?

● Once primary node is down, the whole replica set will be marked as fail and can't do any operation on it until the other nodes vote and elect a new primary node.

● During failover, any write operation not committed to whole replica set will be rolled back

Simple replica set configuration

mkdir -p /tmp/db01mkdir -p /tmp/db02mkdir -p /tmp/db03

mongod --replSet test --port 29001 --dbpath /tmp/db01mongod --replSet test --port 29002 --dbpath /tmp/db02mongod --replSet test --port 29003 --dbpath /tmp/db03

Simple replica set configuration

mongo localhost:29001

Another way to config replica set

rs.initiate()rs.add("localhost:29001")rs.add("localhost:29002")rs.add("localhost:29003")

Extra options for setting replica set

● arbiterOnly○ Arbiter nodes don't receive data, can't become

primary node but can vote.● priority

○ Node with priority 0 will never be elected as primary node.

○ Higher priority nodes will be preferred as primary○ If you want to force some node become primary

node, do not update node's vote result, update node's priority value and reconfig replica set.

● buildIndexes○ Can only be set to false on nodes with priority 0 ○ Use false for backup only nodes

Extra options for setting replica set

● hidden○ Nodes marked with hidden option will not be

exposed to MongoDB clients.○ Nodes marked with hidden option will not receive

queries.○ Only use this option for nodes with usage like

reporting, integration, backup, etc.● slaveDelay

○ How many seconds slave nodes could fall behind to primary nodes

○ Can only be set on nodes with priority 0○ Used for preventing some human errors

Extra options for setting replica set

● voteIf set to 1, this node can vote, else not.

Change primary node at runtime

config = rs.conf()config.members[1].priority = 2rs.reconfig(config)

What is sharding?

Name Value

Alice value

Amy value

Bob value

: value

: value

: value

: value

Yoko value

Zeus value

A value

to value

F value

G value

to value

N value

O value

to value

Z value

MongoDB sharding architecture

Elements of MongoDB sharding cluster

● Config ServerStoring sharding cluster metadata

● mongos RouterRouting database operations to correct shard server

● Shard ServerHold real user data

Sharding config - config server

● Config server is a MongoDB instance runs with --configsrv option

● Config servers will automatically synced by mongos process, so DO NOT run them with --replSet option

● Synchronous replication protocol is optimized for three machines.

Sharding config - mongos Router

● Use mongos (not mongod) for starting a mongos router

● mongos routes database operations to correct shard servers

● Exmaple command for starting mongosmongos --configdb db01, db02, db03

● With --chunkSize option, you could specify a smaller sharding chunk if you're just testing.

Sharding config - shard server

● Shard server is a MongoDB instance runs with --shardsvr option

● Shard server don't need to know where config server / mongos route is

Example script for building MongoDB shard cluster

mkdir -p /tmp/s00mkdir -p /tmp/s01mkdir -p /tmp/s02mkdir -p /tmp/s03

mongod --configsvr --port 29000 --dbpath /tmp/s00mongos --configdb localhost:29000 --chunkSize 1 --port 28000mongod --shardsvr --port 29001 --dbpath /tmp/s01mongod --shardsvr --port 29002 --dbpath /tmp/s02mongod --shardsvr --port 29003 --dbpath /tmp/s03

Sharding config - add shard server

mongo localhost:28000/admin

db.runCommand({addshard: "localhost:29001"})db.runCommand({addshard: "localhost:29002"})db.runCommand({addshard: "localhost:29003"})

db.printShardingStatus()db.runCommand( { enablesharding : "test" } )db.runCommand( {shardcollection: "test.shardtest",key: {_id: 1}, unique: true})

Let us insert some documents

use test

for (i=0; i<1000000; i++) {db.shardtest.insert({value: i});

}

Remove 1 shard & see what happens

use admindb.runCommand({removeshard: "shard0002"})

Let's add it backdb.runCommand({addshard: "localhost:29003"})

Pick your sharding key wisely

● Sharding key can not be changed after sharding enabled

● For updating any document in a sharding cluster, sharding key MUST BE INCLUDED as find spec

EX:sharding key= {name: 1, class: 1}db.xxx.update({name: "xxxx", class: "ooo},{..... update spec})

Pick your sharding key wisely

● Sharding key will strongly affect your data distribution model

EX:sharding by ObjectIdshard001 => data saved 2 months agoshard002 => data saved 1 months agoshard003 => data saved recently

Other sharding key examples

EX:sharding by Usernameshard001 => Username starts with a to kshard002 => Username starts with l to rshard003 => Username starts with s to z

EX:sharding by md5completely random distribution

What is Mapreduce?

● Map then Reduce● Map is the procedure to call a function for

emitting keys & values sending to reduce function

● Reduce is the procedure to call a function for reducing the emitted keys & values sent via map function into single reduced result.

● Example: map students grades and reduce into total students grades.

How to call mapreduce in MongoDB

db.xxx.mapreduce(map function,reduce function,{out: output option,query: query filter, optional,sort: sort filter, optional,finalize: finalize function,.... etc

})

Let's generate some data

for (i=0; i<10000; i++){db.grades.insert({

grades: {math: Math.random() * 100 % 100,art: Math.random() * 100 % 100,music: Math.random() * 100 % 100

}});

}

Prepare Map function

function map(){for (k in this.grades){

emit(k, {total: 1, pass: 1 ? this.grades[k] >= 60.0 : 0, fail: 1 ? this.grades[k] < 60.0 : 0, sum: this.grades[k], avg: 0});

}}

Prepare reduce function

function reduce(key, values){result = {total: 0, pass: 0, fail: 0, sum: 0, avg: 0};values.forEach(function(value){

result.total += value.total;result.pass += value.pass;result.fail += value.fail;result.sum += value.sum;

});return result;

}

Execute your 1st mapreduce call

db.grades.mapReduce(map, reduce, {out:{inline: 1}}

)

Add finalize function

function finalize(key, value){value.avg = value.sum / value.total;return value;

}

Run mapreduce again with finalize

db.grades.mapReduce(map, reduce, {out:{inline: 1}, finalize: finalize}

)

Mapreduce output options

● {replace: <result collection name>}Replace result collection if already existed.

● {merge: <result collection name>}Always overwrite with new results.

● {reduce: <result collection name>}Run reduce if same key exists in both old/current result collections. Will run finalize function if any.

● {inline: 1}Put result in memory

Other mapreduce output options

● db- put result collection in different database

● sharded - output collection will be sharded using key = _id

● nonAtomic - partial reduce result will be visible will processing.

MongoDB backup & restore

● mongodumpmongodump -h localhost:27017

● mongorestoremongorestore -h localhost:27017 --drop

● mongoexportmongoexport -d test -c students -h localhost:27017 > students.json

● mongoimport mongoimport -d test -c students -h localhost:27017 < students.json

Conclusion - Pros of MongoDB

● Agile (Schemaless)● Easy to use ● Built in replica & sharding● Mapreduce with sharding

Conclusion - Cons of MongoDB

● Schemaless = everyone need to know how data look like

● Waste of spaces on keys● Eats lots of memory● Mapreduce is hard to handle

Cautions of MongoDB

● Global write lock○ Add more RAM○ Use newer version (MongoDB 2.2 now has DB level

global write lock)○ Split your database properly

● Remove document won't free disk spaces○ You need run compact command periodically

● Don't let your MongoDB data disk full○ Once freespace of disk used by MongoDB if full, you

won't be able to move/delete document in it.