MongoDB Days Silicon Valley: MongoDB Support Cases: The Blockbusters

MongoDB Support Cases: The Blockbusters

Daniel Coupal Technical Services Engineer

Palo Alto

3

Support Team

4

Goal of the Presentation

•  Remember 3 things from this presentation •  Make 1 change in your application or

deployment •  Good return on your hour spent in this

presentation

5

#1 - “Toto, I've a feeling we're not in Kansas anymore”

6

#1 - “Toto, I've a feeling we're not in Kansas anymore”

ü The Wizard of OZ (1939)

7

#1 – The NoSQL World is different

8

#1 – The NoSQL World is different

Relational World NoSQL World 1 - Model the data 1 - Define the access to the data 2 - Write the queries to access the data

2 - Model the data

9

#2 - “Houston, we have a problem”

10

#2 - “Houston, we have a problem”

ü Apollo 13 (1995)

11

#2 – Have a Test Environment

•  Best –  identical cluster –  restore production data in test/staging/QA

environment

•  Acceptable –  less shards with identical hardware

•  Less desirable –  smaller hardware –  non representative data

12

#3 – “Elementary, my dear Watson”

13

#3 – “Elementary, my dear Watson”

ü The Adventures of Sherlock Holmes (1905, 1921, 1939, 1984)

14

#3 – Look for Clues of Bad Queries (slow queries in mongod log)

•  2.6 format •  2015-09-23T07:06:15.465-0400 [conn21987] query mydb.coll1

query: { tr: { $lte: "2014-08-24-day" }, a: "", b: "min" } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:13798085 nscannedObjects:13798085 keyUpdates:0 numYields:267371 locks(micros) r:105591398 nreturned:0 reslen:20 812656ms

•  3.0 format •  2015-10-15T19:48:15.095+0000 I QUERY [conn1844] query

mydb.coll2 query: { $readPreference: { mode: "primary" }, $query: {key: "ZrzfmfFlUV3i8iARTyXWD0Qxa8I=" } } planSummary: COLLSCAN ntoskip:0 nscanned:0 nscannedObjects:83256 keyUpdates:0 writeConflicts:0 numYields:650 nreturned:0 reslen:20 locks:{ Global: { acquireCount: { r: 1302 } }, Database: { acquireCount: { r: 651 } }, Collection: { acquireCount: { r: 651 } } } 121ms

15

#3 – Look for Clues of Bad Queries

•  mtools for plots and stats on queries

16

#3 – Look for Clues of Bad Queries

•  CloudManager –  Suggested indexes

17

#4 - “Round up the usual suspects”

18

#4 - “Round up the usual suspects”

ü Casablanca (1942)

19

#4 – Settings in Production Notes

•  Hardware –  Using NAS –  Using other FS than xfs on Linux for Wired Tiger –  Using other FS than ext4 or xfs on Linux for MMapV1 –  Maxed out spinning disks

•  OS Settings –  ulimits too low –  too high readahead for disks –  Transparent Huge Pages on –  NUMA on

20

#4 – Settings in Production Notes

•  Tools –  Production notes –  Mdiag (in Github)

21

#5 - “You’re gonna need a bigger boat”

22

#5 - “You’re gonna need a bigger boat”

ü Jaws (1975)

23

#5 – Appropriate Hardware for OpsManager

•  Correctly sizing right at the beginning –  Consulting package –  Open a Commercial Support ticket

•  Beware of changes in the initial estimates

24

#6 - “Bond, James Bond”

25

#6 - “Bond, James Bond”

ü Dr. No (1962)

26

#6 - Cloud/Ops Manager Agents always initiate communications

db.movies.find({“series”:”James Bond”}) .sort({“date”:1}) .limit(1)

27

#6 - Cloud/Ops Manager Agents always initiate communications

Agent Hi, I am part of the 007 group.

Server Go collect information about the hosts goldfinger, scaramanga and doctorno.

Agent Here are their status and metrics for goldfinger and scaramanga.

Agent doctorno is now permanently unreachable.

28

#7 – “You've got to ask yourself one question: 'Do I feel lucky?’ “

29

#7 – “You've got to ask yourself one question: 'Do I feel lucky?’ “

ü Dirty Harry (1971)

30

#7 - Knowledge is King

•  Use the right resources –  MongoDB.com/presentations –  Conferences: MongoDB Days/MongoDB World –  Consulting packages –  Commercial Support –  Free Support (Google groups and Stack Overflow) –  Training classes

Conclusion

32

Take Away

•  Remember 3 things from this presentation •  Make 1 change in your application or

deployment •  Good return on your hour spent in this

presentation

Questions… “You talking to me?”

Thank you and “May the Force be with you”