Mongo db pefrormance tuning with MMS

MongoDB Performance Tuning with MMS

Presentation outline

By Enteros CTO

Ron Warshawsky

[email protected] 408-207-8408

Enteros, Inc.

mailto:[email protected]

MongoDb

2014-03-13 Enteros, Inc.

Overview

What Is MMS MongoDB Management Service is a tool used to determine the health of a system and identify the root cause of performance issues. We will segment this presentation into two parts

Defining Key Metrics

Improving Performance

MongoDb


Defining Key Metrics

We will define two types of metrics here:

MMS Metrics

Log File Metrics

MongoDb


MMS Metrics

By examining these key metrics you can very quickly get a good picture of what is going on inside a MongoDB system and what computing resources (CPU, RAM, disk) are performance bottlenecks. Below is an outlook of the MMS metrics.

PF/OP (Page Faults/Opcounters)

CPU Time (IOWait and User)

Lock Percent and Queues

MongoDb


MMS Metrics

PF/OP (Page Faults/Opcounters)

Between 5 and 10 page faults per second (left) compared to more than 4,000 operations per second (right). A PF/OP of 0.001 (5 / 4000) is close enough to zero to classify as a low disk I/O requirement.

If PF/OP is… near 0 – reads rarely require disk I/O near 1 – reads regularly require disk I/O greater than 1 – reads require heavy disk I/O

MongoDb


MMS Metrics

CPU Time (IOWait and User)

CPU graphs from two different instances: One experiencing high CPU IOWait (left) and the other experiencing high CPU User (right).

The CPU Time graph shows how the CPU cores are spending their cycles. CPU IOWait reflects the fraction of time spent waiting for the network or disk, while CPU User measures computation. Note that to view CPU Time in MMS, you must also install munin.

MongoDb


MMS Metrics


Lock fluctuating with daily load (left) and corresponding queues (right)

Lock Percent and Queues tend to go hand in hand — the longer locking operations take, the more other operations wait on them. The formation of locks and queues isn’t necessarily cause for alarm in a healthy system, but they are very good severity indicators when you and your app already know things are slow.

MongoDb


MMS Metrics


Lock fluctuating with daily load (left) and corresponding queues (right)

The CPU Time graph shows how the CPU cores are spending their cycles. CPU IOWait reflects the fraction of time spent waiting for the network or disk, while CPU User measures computation. Note that to view CPU Time in MMS, you must also install munin.

MongoDb


Log File Metrics

In addition to key MMS metrics we also use MongoDB log files to discuss for analysis.

nScanned

ScanAndOrder

nmoved

MongoDb


Log File Metrics

In addition to key MMS metrics we also use MongoDB log files for analysis and discuss here.

nScanned

ScanAndOrder

nmoved

MongoDb


Log File Metrics

nScanned:

1. The number of objects the database examines to service a query. This counts either documents or index entries, any of which may need to be obtained from disk. 2. It is a good measure of the cost of an operation both in terms of I/O, object manipulation, and memory use. 3. Ideally nscanned is not larger than nreturned, the number of results returned. Also compare nscanned to the size of the collection. If nscanned is large, it is usually because an index is not being used, or that the index was not selective enough. 4. Moving large numbers of documents into memory for these operations is a common cause of page faults and CPU IOWait.

MongoDb


Log File Metrics

scanAndOrder: 1. This is true when MongoDB performs an in-memory ordering operation to satisfy the sort/orderby component of a query. For best performance, all sort/orderby clauses should be satisfied by indexes. Sorting results at query time can be inefficient (as compared with using an index that already contains documents in the desired sort order), and can even block other operations if done while holding the write lock, as with an update or findAndModify. Because sorting requires computation, CPU User time can be expected.

MongoDb


Log File Metrics

nmoved:

Indicates the number of documents that moved on disk during the operation. The higher “nmoved”, the more intensive the operation. This is because when moving a document: the new document location must be in memory, the document must be copied, the old document’s location must be cleared, and index entries must be updated to point to the new location.

MongoDb




correctly determining whether one of these metrics is high or low depends on their values relative to each other. Here are some rules of thumb. • PF/OP is high if it is greater than 0.25

(indicates that your instance is page-faulting for roughly 25% of operations)

CPU IOWait or CPU User is high if • It is greater than 50% or • It is more than twice as large as the next

highest CPU metric

There are eight possible overall assessments given these three metrics.

MongoDb



Unbound

This means that either this database isn’t being used, or it is operating optimally. Instead of relaxing, consider creating test data and a script to load-test your queries.

MongoDb



CPU-Bound

In this case, page faults are not severe enough to prompt any appreciable IOWait. As long as the CPU isn’t idle, it’s somewhat expected that User time will appear the highest. If locking and queueing are occurring, you’re probably seeing in-memory table scans and scanAndOrder operations that are not bottlenecking due to an abundance of RAM or a lack of load. Run through the following steps: • Check your logs for operations with high nscanned or scanAndOrder during periods

of high lock/queue, and index accordingly. • Check your queries for CPU-intensive operators like $all, $push/$pop/$addToSet, as

well as updates to large documents, and especially updates to documents with large arrays (or large subdocument arrays).

• Obtain more and/or faster CPU cores. Importantly, if your database is write-heavy, keep in mind that only one CPU per database can write at a time (owing to that thread holding the write lock).

http://blog.mongolab.com/2013/04/thinking-about-arrays-in-mongodb/

MongoDb



Disk-Bound

if you’re experiencing trouble or if this sort of activity is coming in bursts, then getting the most out of your current disk is about finding the source of I/OWait. So, even though the ratio of PF/OP is low, we’re interested in actual Page Faults, which could still put a noticeable drag on performance. Let’s look at some sources of page faults: • Page faults from insert/update operations – The write load to a database results in

page faults as fresh extents are brought into memory to handle new or moved documents. Adding RAM will not alleviate this cost, so consider faster disks, like SSDs, or sharding.

• Page faults from a shifting working set – These are the page faults that can be prevented with additional RAM or with more efficient indexing.

• Page faults from MongoDB’s internal operations – These are few in number and cannot be prevented.

MongoDb



Disk-Bound: Continued

If you’ve accounted for or the majority of your page faults, the remaining I/OWait in the system should be coming from flushing the journal file and the background synchronization of memory-mapped data to disk. Check writeToDataFilesMB on the Journal Stats graph and the Background Flush Avg graph to see if spikes match the elevated IOWait. Upgrade your disks and/or look into getting your journal on a separate volume than your data. You can also separate databases using the directoryperdb mongod option. The last resort before sharding that youshould be verifying r data model is optimized for writes. Check for collections with a large number of indexes, especially multi-key indexes, and look for alternatives. Check your logs for the “nmoved” flag. If you find a lot of documents are being moved during updates, examine the usePowerOf2Sizes setting for those collections. When it comes to write-friendly data models, look for anything that reduces the number of pages (index or data) that need to be modified when documents are inserted or updated.

MongoDb



CPU/Disk-Bound

Treat elevated IOWait and User times as independently high, and consider each of the previous two sections. We don’t see this combination a lot but it is definitely possible, such as in the case of a high-RAM, write-heavy cluster that is missing an index. If you are experiencing locks and queues, use them to guide the time ranges you focus on.

MongoDb



RAM-Bound

If you are running high PF/OP without CPU IOWait, then: • your operation counts are relatively low (indicating little to no operational activity). • you have fantastic disks that are covering up for what would otherwise be a “RAM-Bound and

Disk-Bound” HHL case. • you just started the database and are “warming” your cache. Wait until memory is fully utilized

and reassess.

If you are experiencing locks and queues, immediately evaluate your indexes to make sure missing indexes are not prompting the need to page fault. Check your logs during periods of high page faulting. Focus on operations with high nscanned values. Increase RAM if necessary, and you should arrive at the coveted LLL case. If you are not locking/queueing, this is technically a sustainable state, but be aware that any growth in operation volume or data size–or even a barely-quantifiable shift in your working data set–could lead to increasing IOWait, locks, and queues.

MongoDb



RAM/CPU Bound

This scenario is similar to the RAM-Bound case above, except that high CPU User activity may be covering up for what would otherwise be a high IOWait. Tackle the CPU portion first, then move towards page faults, in the following steps: 1. Look for nscanned and scanAndOrder activity. On high-write clusters, consider the sheer number

of indexes in your write-collections. Maintaining indexes during inserts and updates can be responsible for this, as can other CPU-intensive operators like $all, $push/$pop/$addToSet, updates to large documents, and updates to documents with large arrays (or large subdocument arrays).

2. Once the CPU User time has decreased, reconsider your case. If you have moved into HHL territory, go to that section. If your CPU User time is still high, beef up your CPU if possible.

3. If your PF/OP is still high, be prepared to move to faster disk as soon as locks and queues begin to occur.

4. Finally, consider sharding.

MongoDb



RAM/Disk Bound

This is possibly the most common problem case, and the solution is fairly straightforward compared to the other cases we’ve dealt with. Your activity is disk-heavy. Index to reduce nscanned, or add RAM either by scaling your machines vertically; or scaling horizontally via sharding.

MongoDb



BoundAllAround

The solution to this protracted situation depends on which of these three metrics is highest. In general, we recommend tackling this as “RAM/Disk-Bound” HHL case first, because lack of RAM is relatively easy to diagnose and many of the solutions for high CPU IOWait can contribute to CPU User improvements. So, check your logs for scanAndOrder and high nscanned numbers and then add indexes to reduce them. Increase RAM if IOWait continues, and then re-evaluate your case. If you still end up here, and if your locks and queues are still a problem, consider increasing your hardware capacity. It could be premature to consider sharding straight from this highly protracted case. Ideally, attempt a resolution that brings you to another case that we’ve explored, and consider sharding in that context.

Enteros


Upbeat High Load Capture

Database Root Cause and Spike Analysis for multi-tiered applications

Enteros UpBeat High Load Capture is an software framework for database problem root cause analysis of Oracle, DB2, SQL Server, MySQL, Sybase and MongoDB database centric multi-tiered applications. High Load Capture user interface visually correlates performance and system load metrics across multiple IT production infrastructure layers. With second-by-second granularity of data analysis, High Load Capture makes analysis possible for the most transient database performance spikes.

Features • Multi-threaded, high-precision performance collection engine • Extensible, dynamically configurable, centrally controlled collection agents • Comprehensive library of collector agents • Cross-tier correlation • Safe, secure agent communication • Load-sensitive collection controller

Enteros


Upbeat High Load Capture

Supported Infrastructure, Database, Application server, OS monitoring

Database Server OS: Linux, Sun Solaris, HP/UX, AIX, Windows Server

Client OS: Windows, Linux

Database: Oracle, Microsoft SQL, IBM DB2, MySQL, Sybase, MongoDB

Application Server: Oracle (BEA) WebLogic, Oracle OAS, JBOSS, IBM WAS

MongoDb


Enteros, Inc

http://www.enteros.com

Enteros is an innovative software company specializing in Performance Management and Load Testing Software for Production Databases - RDBMS and NOSQL/Big Data

Enteros solutions enable IT professionals to identify and remediate performance problems in business-critical databases with unprecedented speed, accuracy

and scope.

Ron Warshawsky; [email protected] 408-207-8408

mailto:[email protected]

Mongo db pefrormance tuning with MMS

Software

mms metrics cpu time

key mms metrics

mms metrics log file

key metrics

mms metrics lock percent

types of metrics

cpu cores

mongodb system