Installing Oracle Database on LDOMS Philippe Fierens
Dec 05, 2014
Installing Oracle Database on LDOMSPhilippe Fierens
About Myself
• Freelance Oracle DBA • I am not a Solaris SysAdmin • Working with Oracle since 2001, Oracle iAS 1.0.2
• Currently working for EurID, • Registry for the .eu TLD • Creator of the fastest open source DNS server on the planet
My Belgian Dark Side:
Agenda
• Project overview : replace an M9000• HW choices• What are LDOMS• LDOM vs ZONES• How we choose to implement it• Problems and solutions
Replace an M9000
• Why• Power hog : cooling + power consumption• Maintenance cost• Not transportable (ok it has wheels but still 1300kg ;-)• Licensing
0,75 core factor ( Ultra Sparc VI ) VS
0,5 core factor ( T4,T5,M5,M6, …)
Our M9000 setupZones to limit number of CPU’s (license)
Requirements
• Biz decision : We stay on SPARC• M – class hardware needed? • Hot-pluggable features needed ?• We have RAC + Data Guard after all
• Can we live with less PCI slots ? • It needs to be “transportable”• Performance ?
Performance
• new T-series ?
• M values oracle provides you eg “T4-2 has a relative performance increase of 1.35 compared to the current M9000”
Is this throughput ? Response time ? what load ?
BUT I PREFER TO SEESEE
Comparing performance
• How to measure that ?• NO Real Application Test license • NO “paid” Load Generator
• What will_not_change • Storage • Network
SLOB to the rescue
• Kevin Clossons Silly Little Oracle Benchmark SLOB V1 and to measure the LIOPS
PERFORMS real world Oracle Database IO
See on
http://kevinclosson.wordpress.com/2012/02/06/introducing-slob-the-silly-little-oracle-benchmark/
What does slob do ?
FOR i IN 1..5000 LOOP v_r := dbms_random.value(257, 10000) ; SELECT COUNT(c2) into x FROM cf1 where custid > v_r - 256 AND custid < v_r;END LOOP;
SELECT COUNT(C2) FROM CF1 WHERE CUSTID > :B1 - 256 AND CUSTID < :B1
Plan hash value: 684126879
---------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |---------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 259 (100)| || 1 | SORT AGGREGATE | | 1 | 132 | | ||* 2 | FILTER | | | | | || 3 | TABLE ACCESS BY INDEX ROWID| CF1 | 256 | 33792 | 259 (0)| 00:00:04 ||* 4 | INDEX RANGE SCAN | I_CF1 | 256 | | 2 (0)| 00:00:01 |---------------------------------------------------------------------------------------
WAIT #18446744071490302944: nam='db file sequential read' ela= 6144 file#=4 block#=651384 blocks=1 obj#=13604 tim=1275446655630
WAIT #18446744071490302944: nam='db file sequential read' ela= 2677 file#=4 block#=653529 blocks=1 obj#=13604 tim=1275446658528
Data Collection
• Collected and Compared SLOB stats • (send me mail if you want complete chart)
• Tested • T4-2 ( 16 cores 8 threads /core) • T5-2 when it came out (32 cores 8 threads / core)• M5000
Some Slob Results1 2 4 8 16 32 64 96 128
0
500
1000
1500
2000
2500
3000
3500
4000
M5K 8 core
T4-1
T4-2 8core ISM
T4-2 8core DISM
T5 8 cores
T5 16 cores
Number of Sessions
Tim
e in
Sec
onds
More Slob Results
SLOB LIOPS totalsessions M5000 8 core T4-8core T5-2-8core T5 16 cores
1 1033,057851 1157,407407 1506,024096 1519,756839
2 1992,031873 2207,505519 2816,901408 2793,296089
4 3868,471954 3861,003861 5347,593583 5319,148936
8 6611,570248 7561,436673 10256,41026 10282,77635
16 8403,361345 11283,49788 15533,98058 20000
32 8565,310493 19207,68307 25723,47267 29739,77695
64 8662,696264 26733,50042 35126,23491 49459,04173
96 32653,06122 61538,46154
128 32636,40999 66736,18352
• We did the tests on M5000 (approx same cpu as M9000)• SLOB LIOPS => (session# * 500.000) / elapsed time
AWR numbers from T5-2
Host Name Platform CPUs Cores Sockets Memory(GB)---------------- -------------------------------- ---- ----- ------- ----------t5play Solaris[tm] OE (64-bit) 128 16 1 512.00
Elapsed: 17.32 (mins) DB Time: 2,179.96 (mins)
Cache Sizes Begin End~~~~~~~~~~~ ---------- ----------
Buffer Cache: 61,440M 61,440M Std Block Size: 8K Shared Pool Size: 8,192M 8,192M Log Buffer: 217,416K
Load Profile Per Second Per Transaction Per Exec Per Call~~~~~~~~~~~~ --------------- --------------- ---------- ---------- DB Time(s): 125.8 11,890.7 0.00 47.81 DB CPU(s): 125.5 11,856.5 0.00 47.67
Redo size: 3,425.6 323,680.0
Logical reads: 15,852,361.5 1,497,873,782.5 Block changes: 8.4 788.8 Physical reads: 0.0 1.4 Physical writes: 1.4 130.6 User calls: 2.6 248.7 Parses: 2.3 213.8 Hard parses: 0.4 37.6W/A MB processed: 0.1 5.0 Logons: 0.1 13.0 Executes: 61,579.7 5,818,606.7 Rollbacks: 0.0 0.0
Transactions: 0.0Avg
wait % DBEvent Waits Time(s) (ms) time Wait Class------------------------------ ------------ ----------- ------ ------ ----------
DB CPU 130,422 99.7library cache: mutex X 144,867 114 1 .1 Concurrencresmgr:cpu quantum 44,990 71 2 .1 Schedulerlibrary cache lock 892 19 21 .0 Concurrenccursor: pin S wait on X 1,689 18 11 .0 Concurrenc
Host CPU (CPUs: 128 Cores: 16 Sockets: 1)
Result
Performance comparison conclusion
• T5-2 gives AMPLE processing power • Fraction of cost• Cap CPU possible ( ZONES or LDOM)
Questions so far?
Virtualizing in Solaris
• Zones aka containers
• LDOM aka OVM for SPARC
* Both technologies can be used to hard partition
Zones
• Global zone • “owns” the hardware and controls the resources
• Non Global zones• Configured and run in global zone • Exclusive access and Shared network access possible, • NO Exclusive access to HBA • Share same kernel as global zone (except branded zones)• Isolated
• Supported to CAP CPU (licensing)
LDOM
• Hypervisor • HW abstraction• presentation virtual devices• Integrated in the HW
• LDOM guests are separated machines, have direct access to CPU and Memory
• Requires CPU that supports this ( sun4v… T x, M5,M6 …)
• Very flexible
Flexibility
• Virtual switches• Virtual networks• Stub devices
• Virtual Disks• Since the IO is virtualized there is a penalty• See later
• Virtual terminal consoles
Roles
• Different types/roles• Control domain : • primary domain • used to connect to HV and manage resources
• I/O domain :• Direct access to phys I/O devices• PCI root complex • PCI card • Regular non virtualized drivers• Limited number of I/O domains depending on platform T5-2 vs
T5-4
Roles (continued)
• Service domain :Provides virtual I/O to guests
Virtual network
Virtual Disk
• Guest domainconsumer of virtual I/O
Requirements and Constraints
• Production isolation• Live migration some guests• Prod + ACC ==> RAC • 4 quad 1 gbit nics per machine • 4 dual 8Gbit HBA• No 10 gbit infrastructure (only “cross-over” cable for live
migration)
Chosen Architecture
• 1 control / IO / service domain :Not a lot of pci slots but we can have “unlimited” virtual nics *
• 1 root domain2 network port for PUBLIC interface
6 network port for PRIVATE interconnect
* Depends on LDC
Root complexes T5-2
Internal HDDs• /pci@300/pci@1/pci@0/pci@2/scsi@0/iport@[1-4]/disk@[0-3] (Controller 0; LSI SAS2308 #1)• /pci@3c0/pci@1/pci@0/pci@2/scsi@0/iport@[1&2]/disk@[4&5] (Controller 1; LSI SAS2308 #2)• CDROM - /pci@3c0/pci@1/pci@0/pci@2/scsi@0/iport@8/cdrom@p3• NET 0 - /pci@300/pci@1/pci@0/pci@1/network@0• NET 1 - /pci@300/pci@1/pci@0/pci@1/network@0,1• NET 2 - /pci@3c0/pci@1/pci@0/pci@1/network@0• NET 3 - /pci@3c0/pci@1/pci@0/pci@1/network@0,1• PCIe 1 - /pci@300/pci@1/pci@0/pci@4• PCIe 2 - /pci@380/pci@1/pci@0/pci@5• PCIe 3 - /pci@380/pci@1/pci@0/pci@6• PCIe 4 - /pci@380/pci@1/pci@0/pci@7• PCIe 5 - /pci@340/pci@1/pci@0/pci@4• PCIe 6 - /pci@340/pci@1/pci@0/pci@5• PCIe 7 - /pci@340/pci@1/pci@0/pci@6• PCIe 8 - /pci@3c0/pci@1/pci@0/pci@7• USB 0 (rear top)
• See note SPARC Platforms: Matrix of Recognized Device Paths (Doc ID 1005907.1)
Setup
Things which we ran into
• RAC and Jumbo Frames• How to provide network to guests• Integration with Symantec netbackup • Impact of Virtual IO on performance
RAC and Jumbo frames
• Make sure MTU size is big enough :• SOLARIS mtu = payload• Juniper switch mtu = payload + overhead
• When not matched, GI root.sh on second node fails ;-(
• Ping is your friend ( the NW admin as well) • ping -s -c 2 -i 192.168.10.1 host-priv1 9000
Network config service domain:
• Traditionally aggr on Service Domain IPMP in the guests.• DLMP• “Combines” IPMP and LACP• No configuration required on Switch • Multiple NIC are combined together• When one nic fails the guests using that NIC repointed to other NIC
DLMP how to
• On Service domain :• Create DLMP interface• Create VNIC to access the machine• Create vswitch on DLMP interface• Create NICs for guests on vswitch
Netbackup Constraints
• Guest no HBA access• service domain = media server• Master server talks to client• We don’t want REAL network traffic• We want to use FCAL to go to VTL
• Backup via stub device to media server
Netbackup Solution
• Create stub device vswitch• Vnic service domain• Vnic Guest
• Master communicates with public address of guest.• on service domain guest is guest.bck.domain (192.168.xx)• On master in /etc/host • Guest.bck.domain = 172.x.x.x
Performance Virtual io
• Tested with orion :• Results :• Random io always faster on Service / ROOT domain • Sequential IO
• Faster on guest then on service domain when >=1MB
• Slower for 8k block on guest
• See my blog for more details in the near futur ;)
SLOB P-IOPS1 2 4 8 12 16 32 48 64
0
5000
10000
15000
20000
25000
30000
35000
root domain
guest
readers
Phy
sica
l IO
per
Sec
ond
Small Recap
• T series can be alternative to M series• Logical Domains : Flexible• Performance penalty with Virtual IO• MEASURE it
Thank you
• Peter De Clerk ( Oracle Belgium)• Kevin Closson (father of SLOB :-) • Karoly Vegh ( Oracle Austria) via twitter• EurID for letting me present this• The three other pragma’s; Iloon, Joel and Jan• Thanks for reviewing @drune / @OsamaOracle
• You can contact me via
be.linkedin.com/in/pfierens/@PFIERENS [email protected]
Pfierens.blogspot.com