This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
AIX 5L/6 Performance Tuning Part I: Tactics for Monitoring Indications of Performance IssuesEarl JewIBM Field Technical Sales Specialist for Power Systems and StorageIBM Regional Designated Specialist - Power/AIX Performance and Tuning400 North Brand Blvd., Suite 700 c/o IBM, Glendale, CA, USA [email protected] (310)251-2907
AIX Virtual Users Group presentation July 29, 2010
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts, Concepts, Considerations, and Tactics
Monitoring AIX – Usage, Meaning and Interpretation– Review component technology of the infrastructure, i.e. proper tuning-by-hardware– Review implemented AIX constructs, i.e. “firm” near-static structures and settings– Review historical/accumulated AIX events, usages, pendings, counts, blocks, etc.– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, topas, etc.
Recognizing Common Performance-degrading Scenarios– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r– Poor ratio of pending I/O waits to iodones in vmstat -s output
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts: Monitoring AIX 5L/6.1 LPARs
Many AIX performance-degrading scenarios can be readily characterized by monitoring AIX dynamically (real-time) as well as cumulatively (ie. vmstat –sv).
By understanding and interpreting the output of mundane AIX commands better&deeper, areas of resource exhaustion, limitation and over-commitment, as well as, resource under-utilization, surplus and over-allocation, can be distinguished.
This presentation focuses on the tactical -- meaning your daily “keyboard awareness”.
This will explain the numbers presented by AIX commands (vmstat, mpstat, iostat, ps, etc.) and formulate the severity of performance issues, if any.
Most cumulative indicators are counts-per-scale over days-uptime.
Many dynamic indicators are comparing ranges&ratios of system resources.
Scaled-definitions define blue/surplus, green/normal, yellow/warning, red/serious and Flashing-Red-with-Sirens/critical status-conditions.
Monitor dynamic AIX behaviors using a 1 or 2 second sampling interval (vs >30secs) Verify a stressful workload exists: “We can’t tune what is not being taxed” Discontinue active efforts when done: “If/when it runs fast enough, we’re tuned” Build with track-able discrete structures: “We can’t tune what can’t be tracked” Monitor spikes,peaks,bursts and burns: “We tune the intensities, not the sleepy-times” Establish dynamic baselines by monitoring real-time AIX behaviors by ranges&ratios Watch AIX behaviors with the goal of characterizing the workload (vmstat –Iwt 2)
AIX Virtual Users Group presentation July 29, 2010
Strategic Tactics: Monitoring/Tactically-Tuning AIX 5L/6.1 LPARs Monitor AIX behaviors with the goal of characterizing the workload Use the workload characterization to guide AIX 5L/6.1 tactical-tuning efforts
Tuning Strategy example 1 Determine points of exhaustion, limitation, and over-commitment Determine surplus resources: CPUcycles, RAM, SAN I/O thruput, etc. Devise tactics to relieve exhaustions by exploiting surplus resources
Tuning Strategy example 2 Study the mechanics of AIX Virtual Memory Management (VMM) Understand the influence of vmo/ioo/no tuning parameters on AIX VMM dynamic behaviors Practice monitoring the behaviors of the AIX VMM mechanisms Recognize-and-Remedy the “bottlenecks” in AIX VMM resources
Tuning Strategy example 3 Exercise&experiment with the various JFS2 mount-options as well as Going Raw Devise ways to characterize I/O patterns in routinely-active RDBMS “tablespaces” Match/place RDBMS “tablespaces” with the best JFS2 mount-options including Going Raw
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts, Concepts, Considerations, and Tactics
Monitoring AIX – Usage, Meaning and Interpretation– Review component technology of the infrastructure, i.e. proper tuning-by-hardware– Review implemented AIX constructs, i.e. “firm” near-static structures and settings– Review historical/accumulated AIX events, usages, pendings, counts, blocks, etc.– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, topas, etc.
Recognizing Common Performance-degrading Scenarios– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r– Poor ratio of pending I/O waits to iodones in vmstat -s output
Paging Space InformationTotal Paging Space: 6016MBPercent Used: 11%
Volume Groups Information============================================================================== rootvg:PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTIONhdisk0 active 511 89 00..00..00..44..45hdisk1 active 511 89 00..14..00..30..45============================================================================== ……
AIX Virtual Users Group presentation July 29, 2010
lsdev # note the count&capacity of the component technology of the LPAR
$ lsdevL2cache0 Available L2 Cacheaio0 Available Asynchronous I/O (Legacy)en0 Available 08-08 Standard Ethernet Network Interfaceen1 Defined 0K-08 Standard Ethernet Network Interfaceen2 Defined 0K-09 Standard Ethernet Network Interfaceen3 Defined 0L-08 Standard Ethernet Network Interfaceen4 Defined 0L-09 Standard Ethernet Network Interfaceen5 Available Standard Ethernet Network Interfaceen6 Available Standard Ethernet Network Interfaceen7 Defined 0L-09 Standard Ethernet Network Interfaceent0 Available 08-08 10/100/1000 Base-TX PCI-X Adapter (14106902)ent1 Available 0K-08 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)ent2 Available 0K-09 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)ent3 Available 0L-08 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)ent4 Available 0L-09 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)ent5 Available EtherChannel / IEEE 802.3ad Link Aggregationent6 Available EtherChannel / IEEE 802.3ad Link Aggregationet0 Defined 08-08 IEEE 802.3 Ethernet Network Interfaceet1 Defined 0K-08 IEEE 802.3 Ethernet Network Interfaceet2 Defined 0K-09 IEEE 802.3 Ethernet Network Interfaceet3 Defined 0L-08 IEEE 802.3 Ethernet Network Interfaceet4 Defined 0L-09 IEEE 802.3 Ethernet Network Interfaceet5 Defined IEEE 802.3 Ethernet Network Interfaceet6 Defined IEEE 802.3 Ethernet Network Interfaceet7 Defined 0L-09 IEEE 802.3 Ethernet Network Interfacefcnet0 Defined 05-08-01 Fibre Channel Network Protocol Devicefcnet1 Defined 07-08-01 Fibre Channel Network Protocol Devicefcnet2 Defined 0A-08-01 Fibre Channel Network Protocol Devicefcnet3 Defined 0D-08-01 Fibre Channel Network Protocol Devicefcnet4 Defined 0G-08-01 Fibre Channel Network Protocol Devicefcnet5 Defined 0H-08-01 Fibre Channel Network Protocol Devicefcnet6 Defined 0I-08-01 Fibre Channel Network Protocol Devicefcnet7 Defined 0J-08-01 Fibre Channel Network Protocol Devicefcs0 Available 05-08 FC Adapterfcs1 Available 07-08 FC Adapterfcs2 Available 0A-08 FC Adapterfcs3 Available 0D-08 FC Adapterfcs4 Available 0G-08 FC Adapterfcs5 Available 0H-08 FC Adapter……
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts, Concepts, Considerations, and Tactics
Monitoring AIX – Usage, Meaning and Interpretation– Review component technology of the infrastructure, i.e. proper tuning-by-hardware– Review implemented AIX constructs, i.e. “firm” near-static structures and settings– Review historical/accumulated AIX events, usages, pendings, counts, blocks, etc.– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, topas, etc.
Recognizing Common Performance-degrading Scenarios– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r– Poor ratio of pending I/O waits to iodones in vmstat -s output
AIX Virtual Users Group presentation July 29, 2010
df –k # review the implemented construction of “firm” AIX structures; observe count-of-inodes per GBs(used) of each application’s data filesystems using Excel
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts, Concepts, Considerations, and Tactics
Monitoring AIX – Usage, Meaning and Interpretation– Review component technology of the infrastructure, i.e. proper tuning-by-hardware– Review implemented AIX constructs, i.e. “firm” near-static structures and settings– Review historical/accumulated AIX events, usages, pendings, counts, blocks, etc.– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, topas, etc.
Recognizing Common Performance-degrading Scenarios– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r– Poor ratio of pending I/O waits to iodones in vmstat -s output
AIX Virtual Users Group presentation July 29, 2010
vmstat –s # Writes to standard output the contents of the sum structure, which contains an absolute count of paging events since system initialization.
address translation faultsIncremented for each occurrence of an address translation page fault. I/O may or may not be required toresolve the page fault. Storage protection page faults (lock misses) are not included in this count.
page insIncremented for each page read in by the virtual memory manager. The count is incremented for page insfrom page space and file space. Along with the page out statistic, this represents the total amount ofreal I/O initiated by the virtual memory manager.
page outsIncremented for each page written out by the virtual memory manager. The count is incremented forpage outs to page space and for page outs to file space. Along with the page in statistic, this representsthe total amount of real I/O initiated by the virtual memory manager.
paging space page insIncremented for VMM initiated page ins from paging space only.
paging space page outsIncremented for VMM initiated page outs to paging space only.
…pages examined by the clock
VMM uses a clock-algorithm to implement a pseudo least recently used (lru) page replacement scheme.Pages are aged by being examined by the clock. This count is incremented for each page examined by the clock.
revolutions of the clock handIncremented for each VMM clock revolution (that is, after each complete scan of memory).
pages freed by the clockIncremented for each page the clock algorithm selects to free from real memory.
AIX Virtual Users Group presentation July 29, 2010
vmstat –s # [continued] Writes to standard output the contents of the sum structure, which contains an absolute count of paging events since system initialization.
backtracksIncremented for each page fault that occurs while resolving a previous page fault. (The new page fault must be resolved
first and then initial page faults can be backtracked.) free frame waits
Incremented each time a process requests a page frame, the free list is empty, and the process is forced to wait while the free list is replenished.
extend XPT waitsIncremented each time a process is waited by VMM due to a commit in progress for the segment being accessed.
pending I/O waitsIncremented each time a process is waited by VMM for a page-in I/O to complete.
start I/OsIncremented for each read or write I/O request initiated by VMM.
iodonesIncremented at the completion of each VMM I/O request.
CPU context switchesIncremented for each processor context switch (dispatch of a new process).
device interruptsIncremented on each hardware interrupt.
software interruptsIncremented on each software interrupt. A software interrupt is a machine instruction similar to a hardware interrupt that
saves some state and branches to a service routine. System calls are implemented with software interrupt instructions that branch to the system call handler routine.
decrementer interruptsIncremented on each decrementer interrupt.
AIX Virtual Users Group presentation July 29, 2010
vmstat –v # [Continued] Writes to standard output various statistics maintained by the Virtual Memory Manager. The -v flag can only be used with the -s flag.
file pagesNumber of 4 KB pages currently used by the file cache.
…numclient percentage
Percentage of memory occupied by client pages. maxclient percentage
Tuning parameter (managed using vmo) specifying the maximum percentage of memory which can be used forclient pages.
client pagesNumber of client pages.
…pending disk I/Os blocked with no pbuf
Number of pending disk I/O requests blocked because no pbuf was available. Pbufs are pinned memory buffersused to hold I/O requests at the logical volume manager layer.
paging space I/Os blocked with no psbufNumber of paging space I/O requests blocked because no psbuf was available. Psbufs are pinned memory buffersused to hold I/O requests at the virtual memory manager
filesystem I/Os blocked with no fsbufNumber of filesystem I/O requests blocked because no fsbuf was available. Fsbuf are pinned memory buffersused to hold I/O requests in the filesystem layer.
client filesystem I/Os blocked with no fsbufNumber of client filesystem I/O requests blocked because no fsbuf was available. NFS (Network File System) andVxFS (Veritas) are client filesystems. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer.
external pager filesystem I/Os blocked with no fsbufNumber of external pager client filesystem I/O requests blocked because no fsbuf was available. JFS2 is an external pager client filesystem. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer.
AIX Virtual Users Group presentation July 29, 2010
netstat –ss # cumulative since last boot; displays only non-zero values (this is an undocumented command-option of netstat)
$ netstat -ssip:
343290037 total packets received257309 fragments received5 fragments dropped after timeout128641 packets reassembled ok98
337255449 packets for this host5871563 packets for unknown/unsupported protocol396 packets not forwardable3083156934 packets sent from this host71209 output datagrams fragmented149137 fragments created2 packets dropped due to the full socket receive buffer2115340 dead gateway detection packets sent
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts, Concepts, Considerations, and Tactics
Monitoring AIX – Usage, Meaning and Interpretation– Review component technology of the infrastructure, i.e. proper tuning-by-hardware– Review implemented AIX constructs, i.e. “firm” near-static structures and settings– Review historical/accumulated AIX events, usages, pendings, counts, blocks, etc.– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, topas, etc.
Recognizing Common Performance-degrading Scenarios– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r– Poor ratio of pending I/O waits to iodones in vmstat -s output
AIX Virtual Users Group presentation July 29, 2010
ps gvww # ascending PID, SIZE:RSS, TIME:PGIN, %CPU, full-syntaxPGIN (v flag) The number of disk I/Os resulting from references by the process to pages not loaded in core
PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND
AIX Virtual Users Group presentation July 29, 2010
investigate by digging deeper # Seek-out and discern anomalies and other atypical events/situations; routinely monitor ps command output for unusual events
$ ps -ef | grep -v "Oct 20"UID PID PPID C STIME TTY TIME CMD
----------- --------------------- ------------------------------------ ------------------ ----------- --------r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se7 2 0 3601032 4779 969 2464 103 0 0 0 1099 35643 12135 86 7 3 4 09:18:218 5 0 3604401 2167 1282 328 293 0 2070 10143 1168 54493 17465 89 10 0 1 09:18:23
Number of kernel threads in various queues averaged per second over the sampling interval. The kthr columns are as follows:
– rAverage number of kernel threads that are runnable, which includes threads that are
running and threads that are waiting for the CPU. If this number is greater than the number of CPUs, then there is at least one thread waiting for a CPU and the more threads there are waiting for CPUs, the greater the likelihood of a performance impact.
– bAverage number of kernel threads in the VMM wait queue per second. This includes
threads that are waiting on filesystem I/O or threads that are blocking on a shared resource, i.e. inode-lock.
– pFor vmstat -I The number of threads waiting on I/Os to raw devices per second. Threads
waiting on I/Os to filesystems would not be included here.
Provides information about the real and virtual memory.– avmThe Active Virtual Memory, avm, column represents the number of active virtual memory
pages present at the time the vmstat sample was collected. It is the sum-total of all computational memory – including content paged-out to the pagingspace. The avmstatistics do not include file pages.
– freThe fre column shows the average number of free memory pages. A page is a 4 KB area
of real memory. The system maintains a buffer of memory pages, called the free list, that will be readily accessible when the VMM needs space. The minimum number of pages that the VMM keeps on the free list is determined by the minfree parameter of the vmocommand.
Information about page faults and paging activity. These are averaged over the interval and given in units per second.
– pi The pi column details the number of pages paged in from paging space. Paging space is the part of
virtual memory that resides on disk. It is used as an overflow when memory is over committed. Paging space consists of logical volumes dedicated to the storage of working set pages that have been stolen from real memory. When a stolen page is referenced by the process, a page fault occurs, and the page must be read into memory from paging space.
Due to the variety of configurations of hardware, software and applications, there is no absolute number to look out for. This field is important as a key indicator of paging-space activity. If a page-in occurs, there must have been a previous page-out for that page. It is also likely in a memory-constrained environment that each page-in will force a different page to be stolen and, therefore, paged out.
– poThe po column shows the number (rate) of pages paged out to paging space. Whenever a page of
working storage is stolen, it is written to paging space, if it does not yet reside in paging space or if it was modified. If not referenced again, it will remain on the paging device until the process terminates or disclaims the space. Subsequent references to addresses contained within the faulted-out pages results in page faults, and the pages are paged in individually by the system. When a process terminates normally, any paging space allocated to that process is freed. If the system is reading in a significant number of persistent pages, you might see an increase in po without corresponding increases in pi. This does not necessarily indicate thrashing, but may warrant investigation into data-access patterns of the applications.
Information about page faults and paging activity. These are averaged over the interval and given in units per second.
– frNumber of pages that were freed per second by the page-replacement algorithm during
the interval. As the VMM page-replacement routine scans the Page Frame Table, or PFT, it uses criteria to select which pages are to be stolen to replenish the free list of available memory frames. The criteria include both kinds of pages, working (computational) and file (persistent) pages. Just because a page has been freed, it does not mean that any I/O has taken place. For example, if a persistent storage (file) page has not been modified, it will not be written back to the disk. If I/O is not necessary, minimal system resources are required to free a page.
– srNumber of pages that were examined per second by the page-replacement algorithm
during the interval. The page-replacement algorithm might have to scan many page frames before it can steal enough to satisfy the page-replacement thresholds. The higher the sr value compared to the fr value, the harder it is for the page-replacement algorithm to find eligible pages to steal.
faults Information about process control, such as trap and interrupt rate. The faults columns are as follows:
– inNumber of device interrupts per second observed in the interval.
– syThe number of system calls per second observed in the interval. Resources are available to user
processes through well-defined system calls. These calls instruct the kernel to perform operations for the calling process and exchange data between the kernel and the process. Because workloads and applications vary widely, and different calls perform different functions, it is impossible to define how many system calls per-second are too many. But typically, when the sy column raises over 10000 calls per second on a uniprocessor, further investigations is called for (on an SMP system the number is 10000 calls per second per processor). One reason could be "polling" subroutines like the select()subroutine. For this column, it is advisable to have a baseline measurement that gives a count for a normal sy value.
– csNumber of context switches per second observed in the interval. The physical CPU resource is
subdivided into logical time slices of 10 milliseconds each. Assuming a thread is scheduled for execution, it will run until its time slice expires, until it is preempted, or until it voluntarily gives up control of the CPU. When another thread is given control of the CPU, the context or working environment of the previous thread must be saved and the context of the current thread must be loaded. The operating system has a very efficient context switching procedure, so each switch is inexpensive in terms of resources. Any significant increase in context switches, such as when cs is a lot higher than the disk I/O and network packet rate, should be cause for further investigation.
cpu Percentage breakdown of CPU time usage during the interval. The cpu columns are as follows:
– us– The us column shows the percent of CPU time spent in user mode. A UNIX® process can execute in either user mode
or system (kernel) mode. When in user mode, a process executes within its application code and does not require kernel resources to perform computations, manage memory, or set variables.
– sy– The sy column details the percentage of time the CPU was executing a process in system mode. This includes CPU
resource consumed by kernel processes (kprocs) and others that need access to kernel resources. If a process needs kernel resources, it must execute a system call and is thereby switched to system mode to make that resource available. For example, reading or writing of a file requires kernel resources to open the file, seek a specific location, and read or write data, unless memory mapped files are used.
– id– The id column shows the percentage of time which the CPU is idle, or waiting, without pending local disk I/O. If there
are no threads available for execution (the run queue is empty), the system dispatches a thread called wait, which is also known as the idle kproc. On an SMP system, one wait thread per processor can be dispatched. The report generated by the ps command (with the -k or -g 0 option) identifies this as kproc or wait. If the ps report shows a high aggregate time for this thread, it means there were significant periods of time when no other thread was ready to run or waiting to be executed on the CPU. The system was therefore mostly idle and waiting for new tasks.
– wa– The wa column details the percentage of time the CPU was idle with pending local disk I/O and NFS-mounted disks. If
there is at least one outstanding I/O to a disk when wait is running, the time is classified as waiting for I/O. Unless asynchronous I/O is being used by the process, an I/O request to disk causes the calling process to block (or sleep) until the request has been completed. Once an I/O request for a process completes, it is placed on the run queue. If the I/Os were completing faster, more CPU time could be used.
– A wa value over 25 percent could indicate that the disk subsystem might not be balanced properly, or it might be the result of a disk-intensive workload.
System configuration: lcpu=24 mem=73728MB ent=12.00
kthr memory page faults cpu time----------- --------------------- ------------------------------------ ------------------ ----------------------- --------r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se
AIX Virtual Users Group presentation July 29, 2010
uptime;vmstat;mpstat # compare and match the meaning of both stats$ uptime ; vmstat -Iwt 2 3 ; mpstat -w 2 107:06AM up 12 days, 10:09, 4 users, load average: 8.97, 9.55, 9.79
System configuration: lcpu=36 mem=79360MB
kthr memory page faults cpu time ----------- --------------------- ------------------------------------ ------------------ ----------- --------r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
AIX Virtual Users Group presentation July 29, 2010
mpstat –w 2 # min:maj:int, (maj=0):int?, cs:ics, int:sysc:pc; monitor ratios to learn what is typical/normal – so you can discern what’s atypical/abnormal
$ mpstat –w 2System configuration: lcpu=12
cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts, Concepts, Considerations, and Tactics
Monitoring AIX – Usage, Meaning and Interpretation– Review component technology of the infrastructure, i.e. proper tuning-by-hardware– Review implemented AIX constructs, i.e. “firm” near-static structures and settings– Review historical/accumulated AIX events, usages, pendings, counts, blocks, etc.– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, topas, etc.
Recognizing Common Performance-degrading Scenarios– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r– Poor ratio of pending I/O waits to iodones in vmstat -s output
AIX Virtual Users Group presentation July 29, 2010
Strategic Thoughts, Concepts, Considerations, and Tactics
Monitoring AIX – Usage, Meaning and Interpretation– Review component technology of the infrastructure, i.e. proper tuning-by-hardware– Review implemented AIX constructs, i.e. “firm” near-static structures and settings– Review historical/accumulated AIX events, usages, pendings, counts, blocks, etc.– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, topas, etc.
Recognizing Common Performance-degrading Scenarios– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r– Poor ratio of pending I/O waits to iodones in vmstat -s output
AIX Virtual Users Group presentation July 29, 2010
67 29-Jul-10
TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
The following are trademarks or registered trademarks of other companies.
* All other products may be trademarks or registered trademarks of their respective companies.
Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:
*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market.
Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.
AIX Virtual Users Group presentation July 29, 2010
DisclaimersNo part of this document may be reproduced or transmitted in any form without written permission from IBM
Corporation.
Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This information could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or program(s) at any time without notice. Any statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
The performance data contained herein was obtained in a controlled, isolated environment. Actual results that may be obtained in other operating environments may vary significantly. While IBM has reviewed each item for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customer experiences described herein are based upon information and opinions provided by the customer. The same results may not be obtained by every user.
Reference in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Any reference to an IBM Program Product in this document is not intended to state or imply that only that program product may be used. Any functionally equivalent program, that does not infringe IBM's intellectual property rights, may be used instead. It is the user's responsibility to evaluate and verify the operation on any non-IBM product, program or service.
THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR INFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g. IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. IBM is not responsible for the performance or interoperability of any non-IBM products discussed herein.
AIX Virtual Users Group presentation July 29, 2010
Disclaimers ContinuedInformation concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
The providing of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to:
IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785USA
IBM customers are responsible for ensuring their own compliance with legal requirements. It is the customer's sole responsibility to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer's business and any actions the customer may need to take to comply with such laws.
IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
The information contained in this documentation is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information provided, it is provided “as is” without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this documentation or any other documentation. Nothing contained in this documentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM (or its suppliers or licensors), or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
AIX Virtual Users Group presentation July 29, 2010
AIX 5L/6 Performance Tuning Part I: Tactics for Monitoring Indications of Performance Issues
Thank YouEarl JewIBM Field Technical Sales Specialist for Power Systems and StorageIBM Regional Designated Specialist - Power/AIX Performance & Tuning400 North Brand Blvd., Suite 700 c/o IBM, Glendale, CA, USA [email protected] (310)251-2907