ID114 - Wrestling the Snake: Performance Tuning 101

© 2012 IBM Corporation

ID114

Wrestling the Snake:Performance Tuning 101

Wes Morgan | Senior Software Engineer

2 | © 2012 IBM Corporation

Agenda■ Why Are We Here?

■ Setting A Reasonable Goal

■ It's a Cycle, Folks

■ Understand Your Environment

■ A word about add on products

■ Take a Picture of “Everyday Life”

■ Common “Red Flags”

■ Thinking about SANs

■ Dealing with the Virtual World

■ A Quick Word About the Operating System


Why Are We Here?

■ Supply and Demand■ “Throwing hardware at it” no longer an option■ Stretching existing systems■ Making upgrades tactical■ Need numbers to justify actions■ Know what to look for


Setting A Reasonable Goal■ We aren't going to Unicorn-and-Rainbow Land

─ Even the best designs can be stressed■ Changing demand■ New initiatives, new software, new users■ Most reasonable goal: Meet peak demand with (some) room to spare


It's a Cycle, Folks

■ You can't do this once and call it finished■ As system load changes, we must adjust and adapt

─ Remember that load can change in volume, footprint, complexity or all of the above■ Revisit performance on a regular basis

─ Know what numbers are important for your enterprise─ Know who supports what in your organization─ Automate reports (monthly, bi-monthly)─ Catch trends before they become problems

■ Revisit performance on any significant change─ New software releases─ Changing OS (or OS version)─ Adding/moving/removing virtual machines─ Networking changes in your enterprise


Understand Your Environment!

■ Know your dependencies─ Directories (Active Directory, LDAP)─ Third-party products (Domino add-ins, Portal adjuncts)─ Back-end components (WCM, DB2, etc.)

■ Know your data flow─ What's handling your data, where, and when?

■ Know your networking─ Are you traversing firewalls?─ Must you use proxy servers?

■ Know where your load lies─ What do clients do?─ What do servers handle?


Understand Your “Everyday Life”

■ Gather baseline performance statistics─ perfpmr (AIX), platform statistics (Domino), perfmon (Windows)

■ First Question: What does “routine” work look like?■ Make sure that your baseline includes at least one “peak use” time

─ Monday morning, 9 am?■ Use this baseline to understand where your system “pushes”■ Identify potential “pressure points”

─ e.g. Peak use of particular facilities, but not yet in the “danger zone”■ Track back to application-layer activities

─ System backups─ Domino agents─ Third-party scripts


Environment, Data Flow and Performance

Application

Authentication

Storage

Third-PartyComponentFirewall


A word about add on products

■ What is an “add on” product?─ Any IBM or third party product or component designed to link into a Domino server.─ Can be a silent killer of resources

■ “Off the shelf” application─ Know the impact before you begin─ Know what the “best practices” are

■ API Tool Kit─ Provide multiple ways to do the same task─ With great flexibility comes great responsibility


A word about add on products – Blackberry Enterprise Server®

■ Research In Motion's answer to mobile connectivity■ Does not run on your mail server but can have a huge impact■ NBES thread count can starve the hand held or overload the mail server


A word about add on products – Applications

■ IBM Lotus Enterprise Integrator (LEI) / Domino Enterprise Connection Services (DECS)

─ Provide the interface to back-end database systems─ Can drive a high disk and network load

■ Extension Manager / NSFHooks─ Test, Test, Test─ Use LotusScript when available


Common “Red Flags” - Disk

■ Disk I/O─ Huge contributor to performance problems─ Disk Queue Length

– # of pending requests (regardless of their size)– Every OS vendor agrees – disk queues longer than 2.0 indicate a bottleneck– Look for both peak queue length and average queue length

─ Disk utilization– Usually expressed as a percentage– Effects of high utilization are obvious– High utilization can contribute to high queue lengths


Common “Red Flags” - Memory

■ Paging─ High paging can indicate RAM starvation─ Also increases disk activity – can send you 'over the edge' with disk latency─ Set paging files to at least 1.5x RAM as baseline

■ Swapping─ High swapping can indicate process thrashing─ Can be triggered by high volume of short-term activity (e.g. scripts, agents)

■ NOTE: No real consensus on “bad numbers” for these─ Examine these in comparison to your healthy “baseline” ─ Look for spikes or high plateaus

■ Virtual servers add to the complexity here!


Common “Red Flags” - CPU■ Looking for repeated spikes or high plateaus■ Easy for one process to effectively hang your system■ May need to go to per-process views to find culprit■ First cut – match against known actions (agents, scripts, backups)


Thinking About SANs

■ They aren't infinite─ Know what shares your SAN (or SAN slice)─ Know general SAN operations (e.g. caching, backups, etc.)─ Remember that another system's I/O can affect your performance

■ They aren't magic─ Applications/OS still see them as a single disk─ Disk queueing and utilization concerns still apply─ Demand can differ drastically among applications


Dealing With The Virtual World■ Regardless of vendor, you're still sharing one set of physical resources■ Still subject to “interference” from other virtual machines■ Still need to provision virtual machines as fully as “real hardware”■ Compare your performance to other virtual machines on the same host■ Be careful with moving virtual machines

─ Pay attention to overall application architecture─ Can lose fault-tolerance/failover

■ Remember, you now have TWO potential bottleneck points─ Within your virtual machine─ The virtual host itself

– Example: Output queue length on VM network interface─ ASK ABOUT OVERCOMMITTING THE VIRTUAL HOST!!!


A Word About OS Patches

■ Be very, very careful■ Some patches can introduce problems all their own■ Use a test environment whenever possible■ Keep abreast of new patches■ Be sure to spot-check performance numbers after patch installation


What About BIOS and Drivers?

■ Keep system BIOS as close to “latest and greatest” as possible─ Some vendors release new BIOS versions almost monthly─ May address/optimize important subsystems─ May resolve particular performance issues

■ Watch for new device drivers as dictated by your performance─ If you're disk-heavy, keep an eye out for SAN/controller drivers─ If you're network-heavy, watch for advanced network drivers

■ Partner with your hardware team!


In Conclusion...

■ No “one right answer” across systems or applications■ Performance tuning driven by your users, your data, your network■ It's a repeating cycle – get used to it■ Partner with network/hardware teams for best effect


Questions & Answers

I'll be in the Meet the Developers Lab all week...

Don't forget the Session Evaluation... THANKS FOR BEING HERE!

Twitter: @wesmorgan1


Legal disclaimer© IBM Corporation 2012. All Rights Reserved.

The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

ID114 - Wrestling the Snake: Performance Tuning 101

Technology

ibm corporationunderstand

ibm corporationa word

ibm corporationenvironment

ibm corporationagenda

ibm corporationits

ibm corporationid114wrestling

ibm corporationsetting

ibm corporationwhy