8/6/2019 Database Performance Tuning Document
1/15
Performance Tuning
Oracle Database
NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
2/15
Performance Tuning
Document Information
Author NIIT Technologies
Reviewer
Approver Richard Van Der Leeden
Effective Date
File
Distribution List
Sl. No Distributed to Issue Date Media Issued by Remarks
1.0 NIIT Transition Team,
Eurostar Transition Team
DataRoom
Change History
Version #
ReleaseDate
SectionsChanged Author Change Description
1.0 23-Jun-2011
NA NA First Release
8/6/2019 Database Performance Tuning Document
3/15
Performance Tuning
Table of Contents
1 Purpose .............................................................................................. 4
2 Audience ............................................................................................. 4
3 References .......................................................................................... 4
4 Glossary & Conventions Used ............................................................... 4
4.1 Acronyms ............................................................................................................. 4
5 Conventions Used ................................................................................ 5
6 Performance Tuning Approaches for Oracle(8.1.6 - 9.2.0.5) on UNIX .... . 5
7 Application Tuning ............................................................................ 7
8 Transition Phase End Sign-off ............................................................ 14
8/6/2019 Database Performance Tuning Document
4/15
Performance Tuning
1 Purpose
The Purpose of this document is to give an overview of finding performance issue in
the database and do the necessary steps to overcome those.
2 Audience
The intended audiences of this document are the delivery / project managementteam and the SME's of Eurostar involved in the Transition process.
3 References
The List of all the referred documents under various sections is given below:
Topic Link
4 Glossary & Conventions Used
4.1 Acronyms
Acronym DescriptionKT Knowledge Transfer/Transition
SME Subject Matter Expert
NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
5/15
Performance Tuning
5 Conventions Used
All text presented in COURIER NEW font is either a script code or a terminaloutput.
6 Performance Tuning Approaches for Oracle(8.1.6 -9.2.0.5) on UNIX
As a database administrator, you will often be confronted with users who say that the response time onthe system is unacceptable.What is unacceptable performance? There are two aspects to this:
Quantifiable performanceImmeasurable performance like user dissatisfaction.
This note will help identify quantifiable performance problems and methods to solve them. When
performance-tuning databases, it is a good to split the problem into parts to facilitate the process.Performance tuning Oracle databases can be divided into three subcategories:
Application tuningRDBMS tuningUNIX system tuning
Each of the above categories has a different maximum possible effect on the performance of your ORACLE database.
Below is the screenshot from Spotlight Performance Management tool being used in eurostar infrastructure. This document covers all the details to tune the various Performance bottlenecks.
NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
6/15
Performance Tuning
NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
7/15
Performance Tuning
7 Application Tuning
Application tuning is by far the most effective aspect for database tuning. But the most important issue isto identify what(module/package functionality, etc) is running slow.
Tuning your SQL statements is the first and foremost investigation since they intervolve both your application and database. To analyze your SQL statements you can set the EVENT 10046 and then tracethe SQL statements:
Add the following line to your init.ora file:
TIMED_STATISTICS = trueRestart the databaseThere are different ways to set the EVENT 10046
If you can run the statement via SQLPLUS then.
SQL >alter session set events '10046 trace name context forever, level 8';SQL > ---run the SQL statement---if the SQL hangs then let this run at least 10 minutes longer than it usually takesSQL > ---If it completes just EXIT OR CTRL +C to stop the query and exit---A trace file will be output in your user_dump_dest(execute a "show parameter user" in SQLPLUS on the server to find the path) .Format this file with TKPROF
$ tkprof sys=no explain =/*username password of the user who executed the queryPlease see NOTE:32951.1 Tkprof Interpretation
Use ORADEBUG to trace a another existing session to get a EVENT 10046 trace file
Please see Note: 1058210.6 HOW TO ENABLE SQL TRACE FOR ANOTHER SESSION USINGORADEBUG
Look at formatted output of trace command and make sure that your SQL statement is using indexescorrectly. Please see Note:163563.1 Resolving Query Tuning Issues
There are links to many more Notes for Query Tuning as well in the above note
RDBMS Tuning
The default init.ora database configuration file is inadequate for large databases.The main tools to use toinvestigate database tuning issues is :
STATSPACK
NIIT Technologies Confidential
http://csp/main/article?cmd=show&type=NOT&id=32951.1http://csp/main/article?cmd=show&type=NOT&id=1058210.6http://csp/main/article?cmd=show&type=NOT&id=1058210.6http://csp/main/article?cmd=show&type=NOT&id=163563.1http://csp/main/article?cmd=show&type=NOT&id=163563.1http://csp/main/article?cmd=show&type=NOT&id=163563.1http://csp/main/article?cmd=show&type=NOT&id=1058210.6http://csp/main/article?cmd=show&type=NOT&id=163563.1http://csp/main/article?cmd=show&type=NOT&id=32951.18/6/2019 Database Performance Tuning Document
8/15
Performance Tuning
HANGANALYZE and SYSTEMSTATE outputs for hanging issuesNotes on Database Tuning available from Metalink
STATSPACK
~~~~~~~~~**Make sure that TIMED_STATISTICS=true in the init.ora
To Install STATSPACK:
SQL> connect / as sysdbaSQL> @?/rdbms/admin/spcreate
You can install STATSPACK objects into its own tablespace to reduce contention.
***Do not install it in the SYSTEM tablespace, create a dedicated tablespace for the statspack schemapreferably on DISKS that are not used by other datafiles
To gather stats:
sqlplus perfstat/perfstatSQL> execute statspack.snap(i_snap_level=>5);wait 20 minsSQL> execute statspack.snap(i_snap_level=>5);
To be able to make comparisons of performance there must be multiple snapshots taken over a period of time.
A minimum of two snapshots are required before any performance haracteristics of theapplication and database can be made.
There are different snap levels as well (see reference Notes below)A good measure is about 20-30 mins apart
To gather a report :
SQL> connect perfstat/perfstatSQL> @?/rdbms/admin/spreport
This will ask you beginning and ending snapshot ids as well as a output file name
For complete reference on STATSPACK: Note:149121.1 Gathering a StatsPack snapshotNote:149113.1 Installing and Configuring StatsPack Package
Note:149124.1 Creating a StatsPack performance report
There are also the following specific Notes that may be helpful:
Note.228913.1 Systemwide Tuning using STATSPACK Reports
NIIT Technologies Confidential
http://csp/main/article?cmd=show&type=NOT&id=149121.1http://csp/main/article?cmd=show&type=NOT&id=149113.1http://csp/main/article?cmd=show&type=NOT&id=149124.1http://csp/main/article?cmd=show&type=NOT&id=228913.1http://csp/main/article?cmd=show&type=NOT&id=149121.1http://csp/main/article?cmd=show&type=NOT&id=149113.1http://csp/main/article?cmd=show&type=NOT&id=149124.1http://csp/main/article?cmd=show&type=NOT&id=228913.18/6/2019 Database Performance Tuning Document
9/15
Performance Tuning
HANGANALYZE and SYSTEMSTATE
for 8.1.7% svrmgrlsvrmgr>connect internal
for 9i and above
% sqlplus "/ as sysdba"
ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME HANGANALYZE LEVEL 3';wait 90 seconds ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME HANGANALYZE LEVEL3';
***EXIT ... then reconnect***
ALTER SESSION SET MAX_DUMP_FILE_SIZE=UNLIMITED;ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 10';
wait 90 secondsALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 10';
wait 90 secondsALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 10';
The output will be in the user_dump_dest directory.
To interpret the hanganalyze and systemstate :
Please see
Note.215858.1 Interpreting HANGANALYZE trace files to diagnose hanging and performance problems
Notes On Metalink to help diagnose Performance issues:~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note.223299.1 Top Oracle 9i init.ora Parameters Affecting PerformanceNote.262946.1 Performance Issues After Increasing WorkloadNote.223730.1 Automatic PGA Memory Managment in 9i
UNIX System AnalysisIt is useful to subdivide UNIX system analysis into three subcategories:
MemoryCPUIO
NIIT Technologies Confidential
http://csp/main/article?cmd=show&type=NOT&id=215858.1http://csp/main/article?cmd=show&type=NOT&id=223299.1http://csp/main/article?cmd=show&type=NOT&id=262946.1http://csp/main/article?cmd=show&type=NOT&id=223730.1http://csp/main/article?cmd=show&type=NOT&id=215858.1http://csp/main/article?cmd=show&type=NOT&id=223299.1http://csp/main/article?cmd=show&type=NOT&id=262946.1http://csp/main/article?cmd=show&type=NOT&id=223730.18/6/2019 Database Performance Tuning Document
10/15
Performance Tuning
1. MEMORY
One of the most common problems when running large numbers of concurrent users on UNIX machines
is lack of memory.In this case, a quick review of UNIX memory management is useful to see what effectlack of RAM can have on performance.
A UNIX machine has virtual memory: the total addressable memory range.Virtual memory is composed of RAM, DISK and SWAP space.Generally, you will want to have the available SWAP space equal to 2 to 3times the RAM.
How does UNIX use SWAP space? It uses two memory management policies: swapping and paging.
Swapping occurs when UNIX transfers an entire process from RAM to a SWAP device. This frees up alarge amount of RAM.
Paging occurs when UNIX only transfers a "PAGE" of memory to the SWAP device. Only a tiny portion of a process might actually be "paged out" to a SWAP device. While swapping frees up memory - it is slower than paging. Paging generally is more efficient but does not allow for large amounts of memory to befreed simultaneously. Most UNIX systems today use a combination of paging and swapping to managememory.
Generally, you will see the following behaviour:
System lightly used: no paging or swapping occurs.System under a medium load: paging occurs as RAM memory runs lowSystem under a very heavy load: paging stops and swapping begins.
When analyzing your UNIX machine, make sure that the machine is not swapping at all and at worstpaging lightly. This indicates a system with a healthy amount of memory available.
To analyze paging and swapping, use the following commands. Commands used in Berkeley UNIX basedsystems will be marked as BSD. Commands used in ATT system V will be marked as ATT.
- vmstat 5 5 (BSD)
procs memory page disk faults cpu
r b w avm fre re at pi po fr de sr d0 s1 d2 s3 in sy cs us sy id
0 0 0 0 1088 0 2 2 0 1 0 0 0 0 0 0 26 72 24 0 1 98
Note: There are NO pageouts (po) occurring on this system. There are also 1088 * 4k pages of free RAMavailable (4 Meg). It is OK and normal to have page out (po) activity. You should get worried when thenumber of page ins (pi) starts rising. This indicates that you system is starting to page.
NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
11/15
Performance Tuning
- pstat -s (BSD)
12112k allocated + 3252k reserved = 15364k used, 37280k available
Note: pstat will also give you the amount of RAM and SWAP currently available on your machine.
- sar -wpg 5 5 (ATT)
09:54:29 swpin/s pswin/s swpot/s pswot/s pswch/s
atch/s pgin/s ppgin/s pflt/s vflt/s slock/s
pgout/s ppgout/s pgfree/s pgscan/s %s5ipf
09:54:34 0.00 0.0 0.00 0.0 12
0.00 0.22 0.22 0.65 3.90 0.87
0.00 0.00 0.00 0.00 0.00
Note: There is absolutely no swapping or paging going on. (swpin,swpot,ppgin,ppgout).
- sar -r 5 5 (ATT)
10:10:22 freemem freeswp
10:10:27 790 5862
This will give you a good indication of how much free swap and RAM you have on your machine.There are 790 pages of memory available and 5862 disk blocks of SWAP available.
There are other platform dependent tools to diagnose memory:
HPUX --> see Note:166490.1 Diagnosing Oracle memory on HP using GLANCEAIX --> see Note:166491.1 Diagnosing Oracle Memory on AIX using SVMONSUN --> see Note:163763.1 Diagnosing Oracle memory on Sun Solaris using PMAP
NIIT Technologies Confidential
http://csp/main/article?cmd=show&type=NOT&id=166490.1http://csp/main/article?cmd=show&type=NOT&id=166490.1http://csp/main/article?cmd=show&type=NOT&id=166491.1http://csp/main/article?cmd=show&type=NOT&id=163763.1http://csp/main/article?cmd=show&type=NOT&id=166490.1http://csp/main/article?cmd=show&type=NOT&id=166491.1http://csp/main/article?cmd=show&type=NOT&id=163763.18/6/2019 Database Performance Tuning Document
12/15
Performance Tuning
CPU
Once you have monitored your systems available memory you will want to make sure the the CPU(s)arenot being overloaded. Here is some general information about how processes get allocated CPU time.UNIX is a multi-processing operating system. That means that a UNIX machine has to manage andprocess multiple user processes simultaneously. UNIX does this in the same way that people wait in lineto buy groceries.
When a process is ready to be processed by a CPU it will be placed on the waiting line or RUN-QUEUE.This is a queue of processes waiting to be run. Obviously there are limits within which one wants to keepthe RUN-QUEUE size. Another factor of interest is the percentage of time the CPU spends in user mode,system mode, or idle mode.
Some commands that determine whether or not there is a CPU resource problem occurring:
- vmstat 5 5 (BSD)
procs memory page disk faults cpu
r b w avm fre re at pi po fr de sr d0 s1 d2 s3 in sy cs us sy id
0 0 0 0 1088 0 2 2 0 1 0 0 0 0 0 0 26 72 24 0 1 98
Note: The CPU is spending most of its time in IDLE mode (id). That means that the CPU is not beingheavily used at all. There are no processes that are waiting to be run (r), blocked (b), or waiting for IO (w)in the RUN QUEUE.
- sar -qu 5 5 (ATT)
10:58:02 runq-sz %runocc swpq-sz %swpocc %usr %sys %wio %idle10:58:07 2.8 100 0 2 4 94
Note: The CPU is spending most (94%) of its time in idle mode.
This CPU is not being heavily used at all. Generally it is alright to have a CPU that is 0% idle as long asthe RUN-QUEUE is not too large. In this case the run queue is 100% occupied (there is always a processwaiting to be run) and there are an average of 2.8 processes waiting tobe run in the RUN-QUEUE.
You want to keep the RUN-QUEUE under 5-6 for a single CPU machine. If the run queue gets any larger that those values you will see some performance degradation. Two solutions to this are:
Obtain a faster processor Use more CPU's.
Avoid overloading your CPU. Response time on your machine will suffer if it is overloaded. Try to keepthe run queue 100% occupied and have less that 6 processes waiting to be run for one CPU. Thischanges as you add more CPU's or a faster CPU. You may also want to avoid the CPU spending most of
NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
13/15
Performance Tuning
its time (more than 50%) in system mode. This may indicate that you are spending too much time inkernel mode servicing interrupts, swapping processes etc.
I/O
The last step in analyzing your UNIX machine is taking a look at IO. After having looked at SQLDBAmonitor to see which datafiles are being used heavily you may also want to take a look at what UNIX saysabout file IO. These commands are for analyzing file IO on file systems.
- iostat -d 5 5(BSD)
sd1 sd3
bps tps msps bps tps msps
1 0 0 .0 4 0 0.0
iostat will display the number of kilobytes transferred per second, the number of transfers per second, andthe milliseconds per average seek. In the example above, both SD1 and SD3 are not used heavily atall.BPS rates over 30 indicate heavy usage of a particular disk. If only one disk shows heavy usage,consider moving some of your datafiles off it or striping your data across several disks.
- sar -d 5 5 (ATT)
09:17:20 device %busy avque r+w/s blks/s avwait avserv
09:17:26 iop0/pdisk010 472.45 1.18 9 107 39.26 512.66
iop0/pdisk000 18.43 2.66 8 132 31.36 24.10
iop0/pdisk020 317.08 1.11 11 165 31.95 294.02
iop0/pdisk021 590.88 1.34 27 518 96.26 219.96
iop0/pdisk040 34.94 1.64 18 113 43.70 19.58
iop0/pdisk041 45.33 1.17 20 79 3.73 22.89
Note: The "-d" option reports activity for each block device.The following is an explanation on the output.
%busy - percentage of time I/O was busyavque - portion of time device was busy servicing a transfer requestr+w/s - average number of requests outstanding during that timeblks/s - number of data transfers from or to device,avwait - number of bytes transferred in 512-byte unitsavserv - average time (in ms) that transfer requests wait idly on queue,and average time to be serviced.
NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
14/15
Performance Tuning
There is a relationship between the number of blocks transferred per second and the average waittime.You can use this to identify which disks are heavily used and which are underutilized.
- sar -b 5 5 (ATT)15:52:57 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
15:53:12 0 2 90 1 2 38 0 0
Note: The "-b" option indicates the overall health of the IO subsystem. The %rcache should be greater than 90% and %wcache should be greater than 60%.
If this is not the case, your system may be bound by disk IO. The sum of bread, bwrit, pread, and pwritgives a good indicator of how well your file subsystem is doing. The sum should not be greater than 40for 2 drives and 60 for 4-8 drives. If you exceed these values, your system may be IO bound.
When analyzing disk IO, make sure that you have balanced the load on your system. Here is a "wish" listof steps for designing a disk layout for Oracle:
-Make sure that your logfiles and archived logfiles are NOT on the same disk as your datafiles. This is abasic safety precaution against disk failure.-Put your files on raw devices.-Allocate one disk for the User Data Tablespace.-Place Rollback, Index, and System Tablespaces on separate disks.-Consider using Raid level 5 disk striping to stripe your datafiles across separate disks.You can also use the above-mentioned UNIX commands to monitor the IO on your system and identifyproblem areas.
Conclusion: There are many ways to approach Oracle performance issues. A structured approach like theone discussed above will allow the system administrator to systematically analyze her system and identifyany problem areas. Once this has been accomplished, measures can be taken to correct problems.Performance is subjective, so find out what is expected.
8 Transition Phase End Sign-off
Transition Phase End Sign Off
Prepared by:
Name:
Approved by:
Name:NIIT Technologies Confidential
8/6/2019 Database Performance Tuning Document
15/15
Performance Tuning
(To be completed and returned by Eurostar )
Accepted by:
Name:
Date : / / ___
NIIT Technologies Confidential