GUOB-Revealing Explaining SPARC Supercluster

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Revealing and Explaining The Oracle SPARC SuperCluster

Eduardo Guedes Systems Sales Consultant Oracle Systems Brazil

Oracle Confidential – Internal/Restricted/Highly Restricted


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Oracle Confidential – Internal/Restricted/Highly Restricted 3


Agenda

Introduction

The Engineered Systems Evolution

SuperCluster Overview

Virtualization on Oracle SuperCluster

Oracle T5-8 SuperCluster Configuration Options

A SuperCluster Configuration Example

1

2

3

4

5


6


About me: • Over 16 years of IT experience including 13 + years as a Certified Oracle

DBA, Administrating Oracle Database on several platforms like Linux, Solaris, AIX, Exadata and SuperCluster.

• Experienced on Oracle Engineered Systems focusing on Exadata and SPARC SuperCluster

• Oracle employee since 2010 and worked under Global Solution Center, ACS and Systems Team as Database Specialist on Oracle HW

• Sun Certified System Administrator for Solaris 8 and 9, OCA/OCP Solaris 11, OCP Oracle Database versions 8, 8i, 9i, 10g and 11g, OCM 11g.

• Detailed profile on http://education.oracle.com/education/otn/EGuedes.htm



Engineered Systems Evolution


2008 2009 2010 2011 2012 2013

Oracle Engineered Systems Timeline

2014

- Exadata

V1

- Exadata V2 - Exadata

X2

- Exalogic

X2

- ODA V1

- T4-4

SuperCluster

- Exadata

X3

- Exalogic

X3

- Exalytics

- Big Data

Appliance

- Exadata X4

- Exalogic X4

- ODA X3-2

- ODA X4-2

(End of 2013)

- Big Data

Appliance X4-2

- Exalytics T5-8

- T5-8

SuperCluster

- M6

SuperCluster

- Virtual Compute

Appliance X4-2

- Oracle Database

Backup Logging

Recovery

Appliance


Exalogic

Elastic

Cloud

Big Data

Appliance Exalytics SuperCluster Exadata Backup,

Logging

and

Recovery

Appliance

Database

Appliance

Virtual

Compute

Appliance

Oracle Engineered Systems


SuperCluster Overview


• SPARC Processors (T5 or M6) for compute of application domains

• Solaris Operating System (Solaris 10 or Solaris 11)

• Infiniband Network, 10Gb Network and FC (Available for configuration)

• Multi-Purpose Storage for application, stage, filesystems and databases

• Exadata Storage for high performance database

• Compatible with Exalogic technology

• Virtualization technologies (PDom, LDom and Zones)

• Redundant and prepared for high-availability requirements.


Oracle T5-8 SuperCluster Components

SPARC T5-8 Compute Nodes

Exadata Storage Servers

InfiniBand Switches

Oracle ZFS Storage ZS3-ES array

• Full Rack

- 2 x SPARC T5-8 compute nodes

• 8 x T5 Processors @ 3.6 GHz

• 2 TB of memory (128 x 16GB)

• 8 x 1.2 TB internal drive (SAS)

• 8 x Sun Dual 10 GbE SFP+ PCIe 2.0 Low Profile Adapter (2 port)

• 8 x Sun Quad Data Rate (QDR) InfiniBand PCIe Low Profile HCA (2 port)

- 8 x Exadata Storage Servers X4-2

- Oracle ZFS Storage ZS3-ES array

- 3 x Sun Data Center InfiniBand Switch 36

- 1 x GbE Management Switch


SuperCluster M6-32 Components

• Solution Components:

– SPARC M6-32

• From 16 to 32 M6 processors and from 8 to 32TB of Memory

– Storage Rack

• 9 * X4-2 Exadata Storage Servers

–Additional capacity can be added with Exadata Storage Expansion racks

• ZS3-ES Storage Appliance

–80TB raw capacity

• 3 x 36-port Infiniband Switches

• GbE Management Switch



• Virtualization is supported without costs

– SPARC M6-32 -> PDom, LDom and Zones

– SPARC T5-8 -> LDom and Zones

• Servers can be separated in Domains (Physical or Logical Domains) for application and database

• Database domains must run on Solaris 11 and access the Exadata Storage

• Application Domains access other Storages and can be used to applications, non-Oracle databases and database versions < 11.2



• For application domains, in order to use Exalogic, the oparating Systems must be Solaris 11

• Application Domains can run on Solaris 10 or Solaris 11

• In order to isolate servers, both database and application domains can run over Zones

• If necessary, other Solaris versions can be used throught Solaris Branded Zones

• The optimized configuration in order to avoid performance issues must be choosen (One option of deployment configuration must be used)



• Zones also have rules, so that, the optimized configuration protects from any kind of performance issues related to misuse/misconfiguration.

• When using Zones, some cores are reserved on global Zones and at least one core must be given to the non-glogal Zones

– At Least 4 cores must be reserved when using Giant, Large or Medium Domains (Domains configuration will be explained later)

– At Least 2 cores must be reserved when using Small Domains (Domains configuration will be explained later)

• Zones and LDOMs are considered hard partitioning for lecensing



• What should you think when using Zones on SSC:

– Plan and create ISCSI LUNs for non-Glogal Zones

– Using anets simplifies the management of the network interfaces

– Plan to use processor sets and resource pools in order to control the available core/threads

– Capped-memory is not supported on SuperCluster Zones

– If using Zones on Database Domains with Zones and there are Zones created for Databases, do not use the Global Zone for Databases

– When using RAC on Zones, plan to create Zones with a similar quantity of Cores/Threads in order to have a good balance of the workload



• What should you think when using Zones on SSC:

– If using RAC databases on Zones, create zones on Database Domains from different T5-8 Servers.

– Calculate the right size for the Exadata Grid Disks in order to not waste Exadata Storage Space

– When configuring the network interfaces for Zones, think that you should configure the Infiniband Partition ID on those in order to communicate with Exadata Storage or ZFS Storage



• Layer 1 Virtualization

– Domains with zero performance overhead

– Any mix of Database Domains and Application Domains

• Oracle Solaris 11 or Oracle Solaris 10 for Application Domains

• Layer 2 Virtualization

– Both Database Domains and Application Domains support Oracle Solaris Zones

App Domain

2

App Domain

1

Zone

SPARC T5-8 Compute Node

DB Domain 1

DB Zone

App Domain

1

DB Domain

2

DB Zone

Zone

Zone DB

Zone

Oracle T5-8 SuperCluster

DB Domain

1

DB Domain

2

DB Zone

Solaris Cluster

Zone

Clusterware

DB Zone Zone

SPARC T5-8 Compute Node


Oracle T5-8 SuperCluster Configuration Options • 1 to 4 LDoms per compute node (half rack)

• 1 to 8 LDoms per compute node (full rack)

• Almost any combination of Database Domains and Application Domains

• Application Domains can boot either Oracle Solaris 10 or Oracle Solaris 11

• SPARC T5-8 Nodes configured independently



Domain Type Domain Footprint on

Oracle SuperCluster

Access to

Exadata

Storage

Servers

Access to Shared Storage

on Included ZFSSA

SAN access via

Optional Fiber

Channel HBAs

Suitable to

host

Exalogic

Software

Zones

Support

Database Domain

Up to four (half rack)

or eight (full rack)

domains per SPARC

T5-8 node

Yes Can be custom configured

post install Supported No Yes

Application Domain

booting Oracle

Solaris 11

Up to four (half rack) or

eight (full rack) domains

per SPARC T5-8 node

No

Configured for optional

Exalogic Elastic Cloud

Software, general

application binaries, and

logfiles

Supported Yes Yes

Application Domain

booting Oracle

Solaris 10

Up to three (half rack)

or seven (full rack)

domains per SPARC

T5-8 node

No

Configured for general

application binaries and

logfiles

Supported No Yes


Oracle T5-8 SuperCluster Configuration Options • Multiple domains of any type supported in a single node

• Flexible Node Layouts

– Nodes can be configured independently

• Note that it makes sense to configure as identical twins in most cases

• CPU/Memory can be resized for each domain after initial install

– Granularity of CPU is 4 cores

– Granularity of Memory is 64GB (half rack) or 128GB (full rack)

– CPU and Memory can be resized independently

• Domains are named as Giant (Only on full RACK), Large, Medium or Small


CPU/Memory Install

Options Domain 1

Default 4 sockets / 1TB

Half-Rack Single-Domain Configurations

Possible Domain Types

Database Domain

Application Domain

(Oracle Solaris 11 only)

SPARC T5-8 Node

Domain

0

HDD4 HDD5

net3 net0

HDD6 HDD7

1 2 3 12 13 14 15

S1 S1

S0 S0 S0

S1

S0

S1

Boot and /u01 Disk

Boot and /u01 Mirror

Mgmt port

(Active)

Mgmt port

(Standby)

HDD0 HDD1 HDD2 HDD3

S0

S1

S0

S1 S1

S0 S0

S1


Half-Rack Two-Domain Configurations

CPU/Memory

Install Options Domain 1 Domain 2

Default 2 sockets / 512GB 2 sockets / 512GB

Option 2 1 socket / 256GB 3 sockets / 768GB

Option 3 3 sockets / 768GB 1 socket / 256GB

Option 4 (Special

case#)

4 Cores / 32GB 60 Cores / 1TB less 32GB

Domain 1 Domain 2

Database Domain Database Domain

Database Domain Application Domain

(Oracle Solaris 11 or Oracle Solaris 10)

Application Domain


Application Domain


SPARC T5-8 Node

Domain 1 Domain 2

net3 net0 net1 net2

0 1 2 3 12 13 14 15

HDD0 HDD1 HDD2 HDD3

S0

S1

S0

S1 S1

S0 S0

S1

HDD4 HDD5 HDD6 HDD7

S0

S1

S0

S1 S1

S0 S0

S1

Boot and /u01 Disk


Mgmt port (Active)

Mgmt port (Standby)


Half-Rack Three-Domain Configurations

CPU/Memor

y Install

Options

Domain 1 Domain 2 Domain 3

Default 2 sockets /

512GB

1 socket /

256GB

1 socket /

256GB

Option 2 1 socket / 256GB 2 sockets /

512GB

1 socket /

256GB

Option 3 1 socket / 256GB 1 socket /

256GB

2 sockets /

512GB

Domain 1 Domain 2 Domain 3

Database Domain Database Domain Database Domain

Database Domain

Application Domain

(Oracle Solaris 11 or

Oracle Solaris 10)

Database Domain

Database Domain

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Application Domain

(Oracle Solaris 11

only)

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

SPARC T5-8 Node

Domain 1 Dom 3 Dom 2

net0 net1 net2 net3 vnet3 vnet1

HDD0 HDD1 HDD2 HDD3

S0

S1

S0

S1

S0

S1

HDD4 HDD5 HDD6 HDD7

S0

S1

S0

S1

S0

S1

0 1 2 3 12 13 14 15

Boot and /u01 Disk


Mgmt port

(Active)

Mgmt port

(Standby)

S0

S1

S0

S1


Half-Rack Four-Domain Configurations

CPU/Memory

Install Options Domain

1

Domain

2

Domain

3

Domain

4

Default 1 socket / 1TB 1 socket / 1TB 1 socket / 1TB 1 socket / 1TB

Domain 1 Domain 2 Domain 3 Domain 4

Database Domain Database Domain Database Domain Database Domain

Database Domain

Application Domain


Oracle Solaris 10)


Database Domain

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Database Domain

Database Domain

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Application Domain

(Oracle Solaris 11

only)

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

S1 S1 S1

S1

S1 S1

SPARC T5-8 Node

Dom 1 Dom 4 Dom 3

net0

Boot and /u01 Disk


Mgmt port

(Active)

Mgmt port

(Standby)

net1 net2 net3 vnet2 vnet0

Dom 2

vnet1 vnet3

HDD0 HDD1 HDD2 HDD3 HDD4 HDD5 HDD6 HDD7

S0 S0 S0 S0

S1 S1

S0 S0 S0 S0

0 1 2 3 12 13 14 15


Full-Rack Two-Domain Configurations

CPU/Memory

Install Options Domain 1 Domain 2

Default 4 sockets / 1TB 4 sockets / 1TB

Option 2 2 sockets / 512GB 6 sockets / 1536GB


Option 4 1 socket / 256GB 7 sockets / 1792GB


Domain 1 Domain 2


Database Domain Application Domain


Application Domain


Application Domain


SPARC T5-8 Node

Domain 1 Domain 2

net0

0 1 2 3 12 13 14 4 5 6 7 8 9 10 11

net2 net3 net1


S0

S1

S0

S1

S0

S1

S0

S1

S0

S1

S0

S1

S0

S1

S0

S1

Boot and /u01 Disk


Mgmt port

(Active)

Mgmt port

(Standby)

15


CPU/Memory Install

Options Domain 1

Default 8 sockets / 2TB

Full-Rack Single-Domain Configurations

Possible Domain Types

Database Domain

Application Domain


SPARC T5-8 Node

Domain

net0

0 1 2 3 12 13 14 4 5 6 7 8 9 10 11

net3

HDD0 HDD1 HDD2 HDD3

S0

S1

S0

S1

S0

S1

S0

S1

HDD4 HDD5 HDD6 HDD7

S1 S1

S0 S0 S0

S1

S0

S1

15

Boot and /u01 Disk


Mgmt port

(Active)

Mgmt port

(Standby)


Full-Rack Four-Domain Configurations

CPU/Memory Install

Options

Domain

1

Domain

2

Domain

3

Domain

4

Default 2 sockets / 512GB 2 sockets / 512GB 2 sockets / 512GB 2 sockets / 512GB

Option 2 1 socket / 256GB 2 sockets / 512GB 2 sockets / 512GB 3 sockets / 768GB

Option 3 3 sockets / 768GB 1 socket / 256GB 2 sockets / 512GB 2 sockets / 512GB

Option 4 3 sockets / 768GB 2 sockets / 512GB 1 socket / 256GB 2 sockets / 512GB

Option 5 3 sockets / 768GB 2 sockets / 512GB 2 sockets / 512GB 1 socket / 256GB

Domain 1 Domain 2 Domain 3 Domain 4

Database Domain Database Domain Database Domain Database Domain

Database Domain

Application Domain


Oracle Solaris 10)


Database Domain

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Database Domain

Database Domain

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Application

Domain

(Oracle Solaris 11

only)

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

Application Domain


Oracle Solaris 10)

SPARC T5-8 Node

Domain 1 Domain 4

net0

0 1 2 3 12 13 14

Domain 2

vnet3

4 5 6 7

Domain 3

vnet2

8 9 10 11

vnet1 vnet0 net2 net3

15

net1

Boot and /u01 Disk


Mgmt port

(Active)

Mgmt port

(Standby)

S1 S1 S1

S1 S1 S1


S0 S0 S0 S0

S1 S1

S0 S0 S0 S0


Full-Rack Eight-Domain Configurations

CPU/Memory

Install Options

Domans 1

through 8

Default 1 socket / 256GB

Domains can be any mix of Database

Domains and Application Domains.

Application Domains can boot either

Oracle Solaris 11 or Oracle Solaris 10.

Note that the first (Control) domain must

boot Solaris 11. This domain is typically a

Database Domain.

SPARC T5-8 Node

Dom 1 Dom 8 Dom 7

net0 net1 net3 vnet2

Dom 2

vnet3


S0 S0 S0 S0 S0 S0 S0 S0

0 1 2 3 12 13 14 15

S1 S1 S1 S1 S1 S1 S1 S1

Dom 3

vnet2

Dom 4

4 5 6 7

Dom 5

vnet2

Dom 6

vnet3

8 9 10 11

vnet0 vnet0 vnet3 vnet0 vnet1 vnet1 net2 vnet1

Boot and /u01 Disk


Mgmt port

(Active)

Mgmt port

(Standby)


A SuperCluster Configuration Examples • Requirements:

– Customer needs Isolation from two different clusters of Oracle Databases

– One application is HA and should be protected by a Cluster in cases of failure

– One application must use Solaris 9 and no HA

– Customer acquires a T5-8 Half SuperCluster

• Solution: – Deployment of the SuperCluster using four Small Domains, 2 database domains and 2

application domains

– One application domain will run Solaris 10 and will have a Solaris 9 branded Zone

– Oracle Cluster will be used to protect the HA application with a Cluster of 2 nodes each node on a different T5-8 server.

– Two RAC databases using 2 cluster of 2 nodes, Grid disks for Data, Reco and DBFS_DG will be created on Exadata for each RAC Database.


A SuperCluster Configuration Examples • Requirements:

– Customer wants to consolidate 8 Databases in a RAC configuration using different Clusters

– Customer wants to use the 4 different environments that must on different machines, isolated from each other, no HA for those environments.

– Customer acquires a T5-8 Half SuperCluster

• Solution: – Deployment of the SuperCluster using one medium Database Domain, Small 2 application

domains

– 8 RAC databases using 8 clusters of 2 nodes, based on Zones, Grid disks for Data, Reco and DBFS_DG will be created on Exadata for each RAC Database.


SuperCluster Reviewing


• Engineered Systems for Oracle Databases

• Engineered Systems for Applications

• Supports Non-Oracle Databases as well

• Supports multiple Oracle database versions not only >=11.2 versions

• Build on the best Oracle Technologies, like: SPARC processors, Exadata Storage, Exalogic SW, Infiniband, etc.

• Supports Virtualization (Pdom, Ldom and Zones)

• Integrates with legacy storage if needed and configured.

• Uses Infiniband Network.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 31

GUOB-Revealing Explaining SPARC Supercluster

Documents

oracle supercluster

oracle andor

oracle hw

administrating oracle

certified oracle dba

sole discretion of oracle

ocp oracle database

systems copyright