Running MongoDB in the Cloud

Running MongoDB in the Cloud

Tony Tam@fehguy

What this Talk is About

Wordnik left the cloud and came back

• What?!?

• Why we left

• Decisions

• Why we came back (and what we did differently)

Who is Wordnik?

•World’s fastest updating English dictionary

• Based on input of text at ~8k words/second

• Word Graph as basis to our analysis

• Synchronous & asynchronous processing

•10’s of Billions of documents in NR storage

•Concept & Meaning Discovery Engine

•> 20M daily REST API calls, billions served

So Why the Detour?

•Architectural Choices

•Business Choices

•Feedback, tooling, infrastructure

•Learning

•Changes in use case

•Progress!

Architecture History

•EC2-based LAMP Stack

• POC (and seed funding)

• A manageable corpus < 1M records

•REST API

• Web + public

• MySQL in master/slave

• ~1B documents

• Operational nightmare

Architecture History

•MongoDB

• First-order MySQL issues solved

• But it got slow…

•Real Servers to the rescue!

• Faster, bigger disks

•MongoDB for Corpus, Structured Data

• Faster Reads + Writes!

• More metal (72GB RAM)

• More cores

• “cold” query from 400ms to < 100

Why Change?

Easy!

•Can’t beat metal…except

• Quick expansion

• Batch jobs/experiments

• Add a datacenter

• Full cluster migration

• The bill for unused capacity

Architectural Mindshift

1. Anything can die, anytime

2. Centralized, redundant state (see point 1)

3. Server performance is *different*

• CPU, I/O, Memory—choose one

• Smart design makes it work!

Architectural Mindshift

•Your software will need to change!

• So will the components you rely on

Your Infrastructure

•Deploying Servers

• Going to need a lot!

•Configuration

•Updates to your software

What about Data?

Cloud Hero

Let’s make this Work!

•MySQL Master Slave

• Take a snapshot (yes, this will block)

• Keep your binlogs!

change master to MASTER_HOST='app1', MASTER_USER='XXXX', MASTER_PASSWORD='XXXX', MASTER_LOG_FILE='app1-relay.0038774', MASTER_LOG_POS=6754205951;

Let’s make this Work!

But…

•Your master is down!

• Quick, promote a slave!

• Point the other slaves to the new master

•As for the clients…

“Well, we never really tried that…”

Better with Mongo

•Easy up, easy down!

• Startup: Sync your data, and announce to clients when ready for business

• Shutdown: Announce your departure and leave

•Replica setsrs.add("db4.wordnik.com:27017");

rs.remove("db1.wordnik.com:27017");

Better with Mongo

But what about Performance?

•Software Design

• It’s slow! (What is *it*?)

• Profile everythingimport com.wordnik.util.perf._

...

def findUser(id:Long): User = {

Profile("UserDao::findUserById", dao.findUserById(id))

}

http://github.com/wordnik/wordnik-oss

But what about Performance?

But what about Performance?

•“It’s the database!”

• What is it?

•Mapping layer

• Mysql (12+ joins) => 50 records/sec

• Mongo JSON POJO => 1000 records/sec

• Mongo DBO POJO => 35,000 records/sec

•How do you know?Profile

it!

It’s Still Slow!

•It’s the index!

• How do you know?

• AHHHHH

It’s Still Slow!

•Balance your B-Tree

• Can't always keep index in ram. MMF "does it's thing"

• Right-balanced b-tree keeps necessary index hot

• If you hit indexes on disk, mute your pager17

15

27

But it’s Still Slow!

•Look at your Schema design

• Design to limit index size/number

• _id is your friend—make it meaningful

• Record size consistency

• Hierarchal Data beware!

• Split documents even in same collection!db.posts.find({_id:/^tony_posts_/})

{_id:"tony_posts_1”, posts:[...]}

{_id:"tony_posts_2”, posts:[...]}

{_id:"tony_posts_3”, posts:[...]}

YOUR app

knows best

Really, it’s STILL slow!

•Your monolithic app/DB won’t scale same on VMs

•Specialize!

• Wordnik uses mSOA

• Data tiers follow service types

• Smaller *everything*

Powered APIswagger.wordnik.com

Really, it’s STILL slow!

•Your monolithic app/DB won’t scale same on VMs

•Specialize!

• Wordnik uses mSOA

• Data tiers follow service types

• Smaller *everything*

Powered APIswagger.wordnik.com

A contract for your clients

Be the Boss of your Data

•Your app *should* be smarter than your DB

• Lots of users?

• Lots of blog posts?

• Lots of images?

• Shard? On what?

•Data dimensionality

• Keep active data hot

• Don’t try to boil the ocean

Cloud Computing + Mongo

•It can work extremely well

• No “Save as Cloud!” menu item

•Shifting constraints

• Optimize for RAM on VM

• Virtual disk => virtual performance

•Be “Deployable”

• Mongo Replica Sets are made for this

Cloud Computing + Mongo

•System Durability

• Design your software for abuse

• Your old design doesn’t apply

• Add APM hooks, now!

•Dissect your app

• Build to micro services with dedicated MongoDB clusters

•Deployment Infrastructure

• Don’t wait until it’s too late

See More

• See more about Wordnik APIs

http://developer.wordnik.com

• Migrating from MySQL to MongoDBhttp://www.slideshare.net/fehguy/migrating-from-mysql-to-mongodb-at-wordnik

• Maintaining your MongoDB Installationhttp://www.slideshare.net/fehguy/mongo-sv-tony-tam

• Swagger API Frameworkhttp://swagger.wordnik.com

• Mapping Benchmarkhttps://github.com/fehguy/mongodb-benchmark-tools

• Wordnik OSS Tools https://github.com/wordnik/wordnik-oss

Questions?