Sizing exadata database_machine

<Insert Picture Here>

Exadata Database Machine Sizing

© 2011 Oracle Corporation – Proprietary and Confidential Last revised – Dec 6, 2011

Goal

• This presentation provides technical guidelines for sizing

Exadata in a pre-sales situation in the following situation

• Customer is re-platforming to Exadata

• Customer is consolidating multiple databases on Exadata

• What it is not:

• New application sizing (e.g. how much Exadata do I need for 30,000

Siebel CRM users)

• Sizing against data warehouse appliances

© 2011 Oracle Corporation – Proprietary and Confidential

Exadata Sizing Methodology

1. Map customer databases to distinct hardware pools

2. Get details of the current databases

• Hardware specs – server, storage

• Database specs – CPU usage, memory utilization, RAC or single

instance, criticality of the database

• Backup requirements

3. Size Exadata for each hardware pool

• CPU, Memory, Storage capacity

• Exadata unique features

• Growth requirements

4. Size DR for production hardware pools


Map Customer Databases to Hardware Pools

• Map customer databases to distinct hardware pools

based on customer’s criteria

• Example mapping criteria are

• Type of database

• mission critical

• standard production

• dev/test databases

• By business unit

• Customer requirements will determine number of

production hardware pools

• For example, pools are marketing-prod, finance-prod, corp-prod,

• Important - Production and dev/test databases be in

separate pools


Current Database Information

• Get legacy database server hardware specs

• Useful for initial worst case sizing estimate

• Easier to get utilization figures using OS tools as

compared to getting individual database information

Existing Database Server Hardware


Server

Name

Model CPU

Type

CPU

Speed

Sockets Cores /

Socket

Threads

/core

True CPU

Peak

Utilization

True CPU

High

Activity Avg

Utilization

Memory Memory

Utilization

ibm01 IBM

P595

Power 7 3.86

GHz

32 8 8 60% 40% 1024

GB

480 GB

• Important – Get High Activity Average utilization which is

average utilization measured during high activity hours (e.g.

first and last hour of trading day)

• Important – Get True CPU utilization (see Appendix) for

multi-threaded CPUs as OS utilities typically understate

CPU utilization

• Get database storage hardware specs

• Important to ensure that storage is sized correctly due to

following

• RAID - Exadata uses ASM normal / high redundancy so extra

raw storage required

• Snapshots – Customers may be using snapshots for backups

and Dev/Test environments

• Requires Exadata to be sized with more storage than the

customers existing storage

• Flash – Newer storage systems use flash as a tiering approach

Existing Storage Hardware


Storage

Server

Model Type RAID Allocated

Storage

Snapshots #disks Disk

RPM

#Flash

Disks

Flash

Size

Connected

Servers

emc01 EMC

DMX

SAN RAID 6 20TB 128 15K ibm01

Existing Database Metrics


• Get information on current database / instances

• Target hardware pool

• Version

• Workload type – OLTP / DW / Mixed

• Database type – RAC, Single Instance

• Criticality - Mission Critical, 24x7, ..

• HA requirements – Standby, RTO, RPO, planned maintenance windows

• Metrics

• Memory utilization of this DB: SGA / PGA targets

• CPU utilization of this DB - Peak and Avg

• Ideally know frequency and timing of peak workloads

• Storage utilization of this DB

• Physical Read/Writes per second

• Space allocated for DB tablespace – allocated and actual usage

• Compression

Existing Production Database Metrics


HW

pool

DB

Name

Version Critical Type #Inst CPU Memory Max

Processes

DB

server Cores Peak Avg SGA PGA

org1 db01 10.2 Yes OLTP 2 12 60% 20% 16 GB 4GB 50 ibm01

DB

Name

Storage Physical IO requests Redo Storage

server Allocated Used Flash Reads Writes Peak rate Total redo / day

db01 124 GB 56 GB 8 GB 400 / sec 100 / sec 1 MB/s 4GB/day emc01

• Estimate CPU/memory usage metrics

• When multiple databases running on single machine – estimate CPU

usage from usage at server level

• Estimate storage capacity/ usage metrics

• Physical reads / writes DB statistic understates actual IO requests as only

considers buffer cache code path

• For 10.2 and later – use “physical read total IO requests” and “physical

write total IO requests” from the AWR reports

Backup Requirements

• Choose between

• Full backup + daily incrementals on Exadata and optionally to tape

• Recommended

• Requires at least 1.5X additional usable space (after mirroring)

• Fastest backup and restore time

• Full backup on non-Exadata disk or to tape only

• Around 25% additional usable space (after mirroring)

• For logs, control file, etc. – verify based on measured redo rates

• Slower backup and restore times

• Size ZFS storage for non-Exadata backup


Sizing Exadata Hardware Pool

Overview – Sizing Exadata Hardware Pool

• For each hardware pool

• Size Exadata hardware first based on CPU requirements to

determine number of database nodes required

• Validate that memory is sufficient on the sized number of nodes

• Increase nodes and/or include memory expansion kits if

additional memory needed

• Validate the storage included with nodes has sufficient capacity

and performance

• Move to a larger rack configuration or add Storage Expansion

Racks if needed

• Fine tune the sizing based on Exadata’s unique software

features

• Factor in growth requirements and add margin for safety


Exadata Hardware Pool - CPU Sizing Details

• Use SPECint to calculate the number of X2-2 or X2-8 cores needed for all

the databases each hardware pool based on the existing peak and High

Activity Average true CPU utilization of the current databases / legacy

server hardware

• Caution – use the true CPU utilization as the OS utilities typically

understate the actual CPU utilization in multi-threaded CPUs

• Adjust number of cores downwards, if competing or replacing slower

CPUs (the SPECint table at the end of the presentation lists the best

case)

• Adjust number of cores upwards by 10%-15% for the clustering

overhead when competing against medium size SMPs (for

applications/workloads requiring large number of CPU cores)

• Note - Large SMPs have scaling issues due to hardware and

software bottlenecks usually requiring no clustering overhead

adjustment.

• If there is a large variation in the number of CPUs required based on peak

and High Activity Average utilization then additional analysis needed



• If there is a large variation between calculated number of

CPUs required based on peak and High Activity Average

utilization

• Additional analysis to see if high activity periods of the various

databases in each hardware pool overlap or not

• If multiple databases and little overlap – then pick number of CPUs

between calculated values based on peak and High Activity Average

utilization

• If no analysis possible or to be conservative, Use higher number based

on peak utilization



• Determine the number of X2-2 or X2-8 database nodes

needed per hardware pool

• Add 1 to the number of database nodes for every 7 active nodes for

HA. Want full processing capacity even if 1 node down for planned or

unplanned maintenance

• Sub-divide into smaller hardware pools if

• no single database larger than recommended max hardware pool size

• Calculated size is greater than the recommended max hardware pool

size

• Recommended hardware pool size - 2 physical Full racks (i.e. 16

X2-2 db nodes or 4 X2-8 db nodes)

• Number of expected OS processes exceeds 20,000 per pool

• Number of instances per node exceeds 128 for X2-2 or 256 for X2-8

• 2000 processes per X2-2 node or 10,000 processes per X2-8 node


Exadata Hardware Pool - Memory Sizing Details

• Validate memory requirements in each hardware pool

• Assume 10GB memory per X2-2 database node and

30GB per X2-8 database node for OS overhead

• Determine the memory requirements for the databases in

each hardware pool using either

• Actual usage from v$pgastat (maximum PGA allocated, max

processes count) and v$sgastat

• Estimated memory usage

Sum of databases (2* PGA_AGGREGRATE_TARGET +

SGA_TARGET) + Max DB Processes * 10 MB

• Choose X2-8 if

• Any single instance requires more than 144 GB memory or total

memory requirements larger the X2-2 max memory capacity


Exadata Hardware Pool – Storage Sizing Details

• Aggregate current storage space used by databases in

each environment

• Factor in compression based on results of the compression

advisor (Note: only table data gets compressed and not the rest

of the database like temp tablespaces, indexes etc)

• Factor in space for FRA

• 25% if no on-disk backups (archive logs, flashback logs, etc.)

• 150% if on-disk backups (full backup, logs, incrementals)

• Calculate raw storage requirements assuming chosen

ASM high redundancy (normal or triple mirroring) for

DATA and FRA

• Triple mirroring of DATA is normally recommended

• Triple mirror FRA for mission critical applications

• Compare to published usable space per machine


Exadata Hardware Pool – Storage Sizing Details

• Validate database storage requirements met for each hardware pool

• Determine # of storage servers (start with rack fraction need to satisfy

CPU/memory requirements)

• If high disk throughput or highest availability – go with High Performance

disks

• Validate storage capacity met. If not, then do the following

• Choose High Capacity disks – if not highest availability or high disk

throughput workloads

• Increase Rack fraction

• Add Storage Expansion Racks – typically if there is lot of “cold” data

and/or to store on-disk backups

• Validate physical read / write IO requirements met

• Important - Make sure that you factor in ASM redundancy when calculating

physical writes (need to double or triple database physical writes based on

whether ASM normal or high redundancy is used)

• Validate Exadata configurations has similar amount of flash as existing

systems


Fine tune sizing of Exadata hardware pools

• Fine-tune sizing of each hardware poool based on Exadata

storage’s unique features

• Smart scans, Storage indexes – offloading processing to storage and

reduction in number of physical I/Os to disk for significant

performance gains for DW style queries

• Flash – increase random read IOPs for better OLTP performance

• Smart Logging – Increase redo logging performance enabling higher

transaction rates


Fine tuning Exadata Sizing

• If CPU requirements slightly above ideal rack fraction (example need 9

X2-2 db nodes but proposing 1.5 X2-2 racks will be too expensive)

• Additional analysis on representative long running queries to estimate

reduced database server CPU usage

• Can reduce CPU requirements by a maximum of 50% for workloads

dominated by queries with large scans (full table / partition scans)

• If physical IO requirements slightly above ideal rack fraction (example

need 35,000 read IO and 30,000 write IOPS (after factoring in triple

mirroring) but X2-2 Full Rack max is 50,000 IOPs)

• Additional analysis of AWR reports to determine how much of the physical

read IO requirements can be satisfied by the Smart Flash Cache

• If lot of reads IOPs are to concentrated portions of the database then

Smart Flash cache will significantly reduce read IOPS requirements

• Write IO requirements can be reduced by moving certain small tablespaces

to flash grid disks

• Use approach only as a last resort and make sure enough flash space

left over for smart flash cache and smart logging


Growth Requirements and Margin of Safety

• Factor in growth requirements and growth strategy

• What is the expected growth rate of CPU, memory, and

storage requirements?

• How many years of future growth should be sized?

• Will hardware be purchased now for future growth or

expanded as required?

• Add in margin for safety


Sizing Exadata DR / Dev / Test

environments

DR Strategy on Exadata

• Local and/or Remote Standby on Exadata

• Highly recommended for critical databases

• Ideally symmetric or identical to production hardware pools –

however can reduce CPU and memory requirements if customer

willing to accept reduced performance during failover

• Define in terms of production hardware pools

• Only size for databases with Standbys

• Scale down factor if willing to not failover some of the load

• Usable storage requirements same as production – but customer

sometimes chooses High Capacity disks

• Note that Primary and DR need to be on separate IB fabrics


Exadata Dev/Test requirements

• Validate customer requirements for Exadata Dev / Test

hardware pools

• Dedicated hardware pool recommended to test patches / application

changes before applying to production

• If supporting mission critical applications, symmetric/identical test

systems to fully simulate production workload for full

performance/capacity planning analysis, HA testing and functional

tests.

• Watch out for storage space as customers may be using snapshots in

their current test/dev environments and that would require more

space in an equivalent Exadata environment

• Dev / Test hardware pools should be on physically separate

InfiniBand fabric than the production hardware pools


Exadata Sizing Output

Exadata Sizing Output

• Details of customer’s current databases

• Correlated production server, storage environments

• Mapping of customer databases to hardware pools

• Proposed Exadata hardware for each of the hardware

pools (e.g. corp-prod, fin-prod, dev/test, DR,…)

• Show available capacity and allocated number of databases,

aggregated CPU, memory, and storage requirements

• List any fine-tuning assumptions made for Exadata’s unique

software features



Representative SPECint

Comparisons


Sun Sparc – SPECint Comparison

Sun

System

Processor CINT2006_rates Equivalent Cores

X2-2 X2-8

M-Series Sparc64 VII (2.75 GHz) 49.1 / 4 cores = 12.3/core 0.38 0.52

M-Series Sparc64 VI (2.4 GHz) 352 / 32 cores = 11/core 0.34 0.47

E25K UltraSparc IV+ (1.95

GHz) 1230 / 144 cores = 8.5/core 0.26 0.36

V890 UltraSparcIV+ (2.1

GHz) 154 / 16 cores = 9.6 /core 0.29 0.41

T5xxx UltraSparc T2 Plus (1.6

GHz) 97 / 8 cores = 12.125/core 0.37 0.51

Note: These are the best case numbers on a per-core basis. X2-2 CPU is 32.6/core and X2-8 CPU is 23.6 / core

http://www.spec.org/cpu2006/results/res2010q1/cpu2006-20091218-09310.html












IBM Power – SPECint Comparison IBM

System

Processor CINT2006_rates Equivalent

Cores

X2-2 X2-8

pSeries Power 7 Eight-Core (3.86

GHz) 652 / 16 cores = 40.8/core 1.25 1.73

pSeries Power6 Dual-Core (5.0 GHz) 2180 / 64 cores = 34/core 1.04 1.44

pSeries Power5+ Dual-Core(2.2 GHz) 1513 / 64 cores = 23.6 /core* 0.62* 0.86*

pSeries Power5+ Dual-Core(1.9 GHz) 314 / 16 cores = 19.6/core* 0.52* 0.72*


Note: IBM does have faster per core numbers for the Power7. But this is based on a quad-core version where they effectively plug in an 8-core chip, turn off 4-cores and run the remaining 4 cores at a faster speed. This is a benchmark special and is not cost effective for the customer.

Note: Power5+ numbers are the older SPEC2000 numbers which need to revised downwards to compare against the SPEC2006 numbers



http://www.spec.org/osg/cpu2000/results/res2006q3/cpu2000-20060724-06700.html



HP Itanium – SPECint Comparison

HP

System

Processor CINT2006_rates Equivalent

Cores

X2-2 X2-8

Integrity Itanium Quad-core 9350

(1.73 GHz) 134 / 8 cores = 16.75/core 0.51 0.71

Integrity Itanium Dual-core 9050

(1.6GHz) 53.9 / 4 cores = 13.5/core 0.41 0.57

Superdome Intel Itanium 2 (1.66 GHz) 1650 / 128 cores = 12.9/core 0.40 0.55

9000

rp4440-8

PA-RISC 8800 Dual-Core

(1 GHz) 37 / 4 cores = 9.25/core* 0.24* 0.34*

9000 Model

4000

PA-RISC 8600 Single Cores

(552MHz) 32.7 / 8 cores = 4.1/cores* 0.11* 0.15*


Note: PA-RISC numbers are the older SPEC2000 numbers which need to revised downwards to compare against the SPEC2006 numbers











Intel – SPECint Comparison

System Processor CINT2006_rates Equivalent

Cores

X2-2 X2-8

HP DL380 G5 Xeon Dual-Core X5270

(3.5 GHz) 90.7 / 4 cores = 22.7 / core 0.70 0.96

HP DL380 G5 Xeon Quad-Core X5365

(3.0 GHz) 116 / 8 cores = 14.5 / core 0.44 0.61


(3.33GHz) 150 / 8 cores = 18.75/core 0.58 0.79


(2.93 GHz) 251 / 8 cores = 31.4/core 0.96 1.33

HP DL580 G5 Xeon Six-Core X7460

(2.66 GHz) 158 /12 cores = 13.2/cores 0.40 0.56










AMD Opteron – SPECint Comparison

System Processor CINT2006_rates Equivalent

Cores

X2-2 X2-8

HP DL185 G5 Opteron Dual-Core 2222

(3.0 GHz) 61 / 4 cores = 15.25/core 0.47 0.65

HP DL385 G5p Opteron Quad-Core 2389

(2.9 GHz) 143 / 8 cores = 17.9/core 0.55 0.76

HP DL585 G5 Opteron Six-Core 8439

(2.8 GHz) 416 / 24 cores = 17.3/core 0.53 0.73

HP DL385 G7 Opteron 12-core 6176

(2.3 GHz) 398 / 24 cores = 16.6/core 0.51 0.70












True Multi-thread CPU

utilization

True Multi-threaded CPU Utilization

• Hardware CPU Threads count as CPUs at the OS level

but provide only incremental benefits

• Example - Intel Nehalem or newer has two threads per

core

• Count second thread as 20% of first thread

• For CPU utilization less than 50% multiply by 1.7

• For CPU utilization over 50% assume 85% plus (util-50%)* 0. 3

• AIX and Solaris have utilities which measure true CPU

utilization

• Use table in next slide to calculate true CPU utilization



Multi-threaded CPU Utilization

CPU True utilization

Intel 5500 and later

If (util < 50%) then

true_util = util * 1.7

else

true_util = 85% + (util – 50) * 0.3

Intel Itanium 2

If (util < 50%) then

true_util = util * 1.7

else

true_util = 85% + (util – 50) * 0.3

UltraSPARC, SPARC64

VI, and SPARC64 VII Max of utilization shown by (corestat, vmstat)

IBM Power Physical core utilization shown by sar

Note – corestat on Solaris SPARC and sar on IBM Power systems provide true core utilization unlike typical OS utilities like vmstat

Sizing exadata database_machine

Technology