Exadata Storage - Availability Best Practices Exadata Storage Availability Best Practices: Beyond Traditional Storage Ashish Ray VP of Product Management Oracle Michael Nowak Consulting

Oracle Exadata Storage Availability Best Practices: Beyond Traditional Storage Ashish Ray VP of Product Management Oracle

Michael Nowak Consulting Member of Technical Staff Oracle

Giri Arunagiri Senior Manager Symantec Robert Hesketh Database Services Manager CCI

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 2

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Abstract

Oracle Exadata’s intelligent storage server eliminates the deficiencies associated with traditional storage. It is simple to deploy, delivers very high performance, and ensures that demanding high-availability service levels are achieved. In this session, Oracle Product Development experts describe relevant features and best practices, how I/O storage bottlenecks are eliminated, how storage failures are handled transparently, how corruptions are detected/repaired without intervention, and how storage migrations/upgrades/expansions are done with zero downtime. These differentiators significantly increase availability and reduce cost, compared to traditional storage. You must attend this session if you are thinking of migrating to Oracle Exadata.


Program Agenda

Traditional Storage Considerations

Exadata Storage: Availability Architecture

Symantec: Exadata in Action

CCI: Exadata in Action


Oracle Database Architecture Logical View Physical View


Database Expectation From Storage

Performance – High throughput, low latency – Predictable performance for varied workload sizes

Scalability – Ability to expand with data growth – Ability to add processing power to accommodate growing workloads

Availability – Storage faults happen – Faults should be transparent / isolated as much as possible


The Anatomy of a Storage Controller

FC

Ethernet Fibre Channel

PCI Express

SAS

Storage controllers/heads are specialized servers running hundreds of thousands of lines of code handling complex operations

Client traffic comes in at 10GbE or 4-16Gb FC

One client request: multiple I/O requests to drives

Writes ack-ed once written to disk or NVRAM

Data flows through many interfaces, each interface can have bugs, add corruptions, etc.

With complex functionality – demand increases on storage resources (compute & bandwidth)

Disks

Host Computer

Client IO

PCI Express

Storage IO Layer

CPU Complex


Performance Considerations

Typically disk shelves are shared between two controllers Storage controllers incredibly busy to with complex storage operations

– Thin provisioning / Storage tiering – Cloning / Snapshots – Mirroring / Deduplication – RAID / Volume management

Increased data growth => increased capacity, but not increased I/O bandwidth and compute power

– Performance actually degrades, unless upfront over-provisioned – May require fork-lift upgrade

Understanding Your Storage Bottlenecks


Introduce flash in your architecture – As a cache or persistent storage

Scale shared storage beyond the two controllers

– Beyond traditional disk reservation systems Make your storage infrastructure application aware

– Process the I/Os in order of importance to the application

Performance Considerations Eliminating Your Storage Bottlenecks


Refer to this session from Oracle Development Team:

– CON8691 - Oracle Database-Aware Flash: Maximizing Performance and Availability for Your Database

– Thursday, Sep 26, 11:00 AM - 12:00 PM

– Moscone South - 236

Performance Considerations Merits of Database-aware Flash


Granular storage nodes, with capacity, processing power and I/O bandwidth Dynamically expand storage

resources to meet business & data center growth (users, apps, files, databases, servers) Scale-out storage software

manages load-balanced and highly available I/O access across the unified storage grid

Performance Considerations: Scale-out Storage

Scaling out storage for increased capacity, compute and I/O bandwidth


Embed application awareness into storage management, examples:

– Filter data at source to reduce I/O load

– Avoid data being backed up to be stored in flash

– Enable better block corruption checks

Performance Considerations: App-Aware Storage


Availability Considerations

Silent corruptions: not detected, not communicated Typically caused by software / firmware bugs Possible antidotes: RAID, checksums, write verify,

disk scrub, … Not sufficient: for example, partial / lost / stray writes

still undetected Deep application knowledge required for adequate

detection and repair

Perils of Silent Data Corruptions


Exadata Storage – Unique Value Proposition

Performance and Availability in One Unified Platform

Integrated Flash

Integrated Scale-out

Integrated Database Awareness

Integrated Data Protection


Program Agenda






Exadata Architecture Complete Database Platform Using Standard Servers for Compute and Storage

Scale-Out Database Servers • 8x 2-socket, or 2x 8-socket Xeon database servers • Oracle Database, ASM, RAC; Linux or Solaris • Standard Ethernet to data center

Scale-Out Intelligent Storage Servers • 2-socket storage servers, Exadata Storage Software • Up to 500 terabytes disk per rack • 56 PCI Flash memory cards per rack

InfiniBand Network • Unified internal connectivity ( 40 Gb/sec )


Exadata Storage Server Building Block • High-performance storage server built from

industry standard components

• 12 disks - 600 GB 15000 RPM High Performance SAS or 3TB 7200 RPM High Capacity SAS

• 2 Six-Core Intel Xeon Processors (E5-2630L) • Dual ported 40 Gb/sec InfiniBand • 4 x 400 GB Flash Cards • Intelligent Exadata Storage Server Software


Exadata Intelligent Storage Grid Data Intensive processing runs in Exadata

Storage Grid – Filter rows and columns as data streams from

disks (168 Intel Cores) Example: How much product X sold last quarter

– Exadata Storage Reads 10TB from disk – Exadata Storage Filters rows by Product & Date – Sends 100GB of matching data to DB Servers

Scale-out storage parallelizes execution and removes bottlenecks


Exadata Storage Differentiators

Storage bottlenecks eliminated and I/O latency reduced

Storage failures handled with extremely low application impact

Corruption detection/repair with no user intervention

Storage upgrades, expansions and migrations with zero downtime


Eliminating Storage Bottleneck: InfiniBand

Storage network – InfiniBand network is pre-

cabled for performance and availability Each InfiniBand link

support 40Gb/s for high throughput

Oracle uses a direct memory access protocol (RDS), reducing CPU consumption and latency

Database Servers

Storage Servers

IB Fabric


Eliminating Storage Bottleneck: ASM

Storage management – ASM intelligently balances

files in a diskgroup with fine or coarse grain striping - eliminates hot spots and reduces IO latency

– ASM rebalance used when a diskgroup becomes imbalanced - most common case is storage configuration change: completely automated with Exadata








Exadata & MAA: Integrated

Not good enough to have component level fault tolerance

What really matters: post failure application impact & proper repair

Exadata and MAA development teams focused on lowering application impact to meet most stringent SLAs

This is a unique value add of engineered systems

We shared this value proposition with the world recently: – Play Video Snippet – Full Video: http://vimeo.com/62754145

http://vimeo.com/62754145�


Handling Disk Failures with Low App Impact

Sub second application impact after disk failure Engineered system benefit: smart flash cache, quick detection and

mirror read, redundancy restored without application impact

0

0.1

0.2

0.3

0.4

Seco

nds

Time

Disk Failure

Response Time


Sub second application impact after flash failure Engineered system benefit: Quick detection and mirror read, quick

repair with resilvering when using Write Back Flash Cache

0

0.1

0.2

0.3

0.4

Seco

nds

Time

Flash Failure

Response Time

Handling Flash Failures with Low App Impact


Sub second application impact after cell failure Engineered system benefit: InfiniBand network leveraged for storage

cell status, preconfigured ASM failure groups ensuring no data loss on a cell failure

0

0.2

0.4

0.6

0.8

1

Seco

nds

Time

Cell Failure

Response Time

Handling Cell Failures with Low App Impact


Up to 5 second application impact after IB network failure Engineered system benefit: network configuration customized and

tested for lowest impact

0

1

2

3

4

Seco

nds

Time

IB Network Failure

Response Time

Handling Storage Network Failures with Low App Impact








Detect / Repair Corruption without User Intervention

Corruptions are rare, but can be crippling

Oracle Database & Exadata are uniquely qualified to detect corruptions that traditional storage cannot

Good copy of data returned from ASM mirror, and bad copy repaired when possible


Auto Corruption Detection / Repair – Example Alert log example. Note this is completely transparent to the application.

Corrupt block relative dba: 0x16400087 (file 89, block 135)

Bad check value found during multiblock buffer read

Data in bad block:

type: 6 format: 2 rdba: 0x16400087 last change scn: 0x0000.b6702b33 seq: 0x1 flg: 0x04 spare1: 0x0 spare2: 0x0 spare3: 0x0 consistency value in tail: 0x2b330601 check value in block header: 0xa07a computed block checksum: 0x3

Reading datafile '+DATA/qs/datafile/c.257.825768683' for corruption at rdba: 0x16400087 (file 89, block 135)

Read datafile mirror ‘DATA_CD_08_CEL13' (file 89, block 135) found same corrupt data (no logical check)

Read datafile mirror ‘DATA_CD_07_CEL14' (file 89, block 135) found valid data

Hex dump of (file 89, block 135) in trace file /u01/app/oracle/diag/… /qs1_ora_60475.trc

Repaired corruption at (file 89, block 135)








Storage Upgrades with Zero Downtime

One simple command to upgrade Exadata storage with no downtime: patchmgr –cells <cell group file> -patch –rolling

Redundancy automatically and efficiently restored


New 11.2.3.3 patchmgr alerts on progress. Example:

Storage Upgrades: Progress Alerts


Exadata Storage Scale-out With Zero Downtime

Demo: Exadata Storage Scale-out


Simplest method, if migrating from similar platform: – Create Exadata based Data Guard Physical Standby – Switchover – DONE!!

Other methods

– Cross Platform Incremental Backups – Oracle GoldenGate – Data Guard Transient Logical Standby (if upgrade involved)

Storage Migration with Minimal Downtime


Program Agenda





37

Exadata High Availability Solutions

Giri Arunagiri Sr. Manager, Symantec

OOW - 2013

Agenda

38

Symantec Authentication Division Overview 1

Why Engineered Systems? Exadata 2

Our Architecture 3

DB High Availability Solutions 4

Exadata Patching 5

OOW - 2013

Symantec Authentication Division Overview • Formerly VeriSign • Acquired by Symantec in August, 2010 • #1 in SSL Certificates issued

– VeriSign, Thawte and GeoTrust brands

• Strong Authentication for network, web and mobile devices • Two Factor Authentication • Public Key Infrastructure ( PKI ) services • User Authentication for the government

39 OOW - 2013

Symantec Sets the Highest CA Policies

OOW - 2013 40

Pioneer in SSL technologies Pioneer in SSL technologies Pioneer in

Worldwide 1 Million web servers worldwide 1 Million web servers worldwide 1 Mil n SSL Certificates Issued Leader in SSL Certificates Issued Leader in SSL Cert

e 2004 100% validation services uptime since 2004 100% validation services up

CA data centers 14 global data centers 13 global CA data centers 14 global

5 billion online OCSP lookups/ day 4.5 billion online OCSP lookups/ day 4.5 bill CSP queries <1 second response for OCSP queries <1 second response for OC

elligence Network Global Intelligence Network/STAR Global Intelligence Networ

blic CA 1st Public CA 1st Public CA 1st Public CA 1st Public CA 1st Public CA 1st Pu

* OSCP: online certificate status protocol

Why Engineered Systems? • Integrated Stack

– Pre-packaged solution with hardware, OS, Clustering and database components

– Integrated and Optimized by Oracle – Faster time to deploy

• One vendor for support – For OS, Clusterware, InfiniBand, Storage and

Database support – Will save us time and effort in complex

performance/outage issues

41 OOW - 2013

Symantec Strong Authentication Solutions now powered by Oracle Exadata & Oracle Active Data Guard

Exadata Enables Systems & Data Consolidation AIX Infrastructure

IBM P6

SSC 10g

IBM P6

GeoTrust 11gR1

IBM P6

Pilot 11g/10g

SRDF

IBM P55A

VIP 10g

VMAX1 VMAX2

CX 0833

IBM P6

LDR

NS 480

ILG - Production

IBM P4

VIP 9i

CX 4445

IBM P5

FDS 10g

CX 3018

BRN - Production

IBM P6

RDR

NS 480

MTV - DR

IBM P5

ECAS / Device 10g

VIP-W 10g

IBM P5

VIP-E 9i

IBM P55A

FDS – Perf 11gR2

CX 0830

CX 3038

IBM P6

FDS – DEV / QA 11gR2

MTV - EDC

IBM P5

IBM P5

Partner 10g Standby

IBM P5

IBM P55A

RAC 11gR1

CX 2915

IBM P6

Dev/QA 11gR1

Waltham – GeoTrust Dev/QA

OOW - 2013

Exadata X3-2 Half Rack (50 TB raw)

West Coast DC QA/Dev

46 Test/Dev databases

East 1

Active Data Guard

East 2

7 production databases – size range from 50GB to 2.5TB

15 standby databases for Local DR

West East Coast DC

Active Data Guard

2 standby databases for Remote DR

15 production databases – size range from 25GB to 500GB





42

http://images.google.com/imgres?imgurl=http://www.rcsscotland.com/images/512px-Padlock.svg_1_.png&imgrefurl=http://www.8bitcollective.com/forums/viewtopic.php?id=7568&usg=__Z7T7yrkq6HXac8U1euseTI-Ecrw=&h=512&w=512&sz=102&hl=en&start=1&tbnid=dCkoPRIz_BqzPM:&tbnh=131&tbnw=131&prev=/images?q=pad+lock&gbv=2&hl=en�


Symantec Strong Authentication Solution: Database Architecture West Coast Data Center

QA/Dev East 1

Active Data Guard *

East 2 West

Active Data Guard *

East Coast Data Center

Prod DB 4 Node RAC

Prod DB 4 Node RAC

Local DR 4 Node RAC

Local DR 4 Node RAC

Remote DR 4 Node RAC Remote DR 4 Node RAC

OOW - 2013

46 Test/Dev databases



7 production databases – size range from 50GB to 2.5TB






43

* Active Data Guard utilized for real-time Authentication Reporting Applications



Database High Availability Solutions Failover and Switchover

Cluster1 Node1

Cluster1 Node2

Cluster2 Node2

Cluster2 Node1

Client Node Client Node

Normal Connections

Node Failover

Primary db Standby db Active

Data Guard

Primary db Standby db

Cluster Switchover

OOW - 2013 44


Exadata Patching

• Quarterly patching • Adhere to Audit and Compliance requirements • Dev -> QA -> DR -> Prod • At least a week gap between each of them • Completely online and transparent to end users

45 OOW - 2013

Exadata Patching – Cell Servers • Each half rack has 7 cell servers • Reboot ILOM to confirm console access • Patch cell servers, which comprises of the following:

– Take it offline – Confirm the grid disk status is offline – Patch cell server – Put it back online

• Completely transparent to databases and end users • ASM resynch occurs transparently, no ASM rebalancing

46 OOW - 2013

Exadata Patching – Database Servers

• Each half rack has 4 db servers •Patching steps

– Take db server out of rotation – Patch it – Put db servers back in rotation – Patch the remaining db servers the same way

•Completely transparent to end users

47 OOW - 2013

Thank you!

Copyright © 2013 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

48 OOW - 2013


Program Agenda





Castleton Commodities International LLC

CCI’s Experience on Exadata September 25, 2013

About CCI: “Castleton Commodities International LLC (CCI) is a leading merchant energy company, CCI is an active participant in the global markets for energy commodities, and operating midstream and upstream energy assets.” - www.cci.com

http://www.cci.com/�





CCI

51 51

DBA Team at CCI

Robert Hesketh, Database Services Manager, Oracle Certified Professional Team of 5 DBAs

Our focus is on more than the Database layer

Manages over 25 databases ranging in size from 128 GB to over 13 TB

Manages over 13 Application Systems (full stack), 3 Database Platforms, 5

Application Server Platforms and over 60 ETL processes

CCI

Primary Data Center Setup - Production Before Exadata

• Six Production 2-node clusters

• Standard Environment Setup: 2 x HP Blade Servers Red Hat 5 Oracle Clusterware\ASM 11gR2 Oracle Database 11gR2 & 10gR2 Flex Fabric CNA for LAN\SAN InfiniBand Interconnect Brocade Silkworm Switches EMC VMAXe Storage System Mix of Solid State, High Performance and High Capacity Disk using FAST

After Exadata • Three Production 2-node clusters • One Exadata X3-2 Quarter Rack

• Exadata Environment Setup: Exadata X3-2 Quarter Rack Oracle Enterprise Linux 5 Oracle Clusterware\ASM 11gR2 Oracle Database 11gR2 High Performance Disk

Note: Second Exadata X3-2 for QA / DR

52

CCI

53 53

Did Exadata Help?

Performance on HP\EMC RAC: CPU Bottlenecks End-of-day Processing at around 2 hours

Performance on Exadata: CPU Bottlenecks alleviated End-of-day Processing at around 1 hour Greater than 30% overall performance increase

Conclusion:

Exadata’s intelligent storage allowed us to offload some CPU processing from the compute nodes

CCI

Assessment Benefits

Meet performance SLAs Out of the box performance allows

CCI resources to focus on additional improvements

Performance capacity of the Exadata allows for consolidation and future cost savings

Standard Exadata Configuration allows for Database High Availability out of the box

Challenges

Database compute node processing capacity and storage capacity are tied If Data Center space available, this issue

addressed by Storage Expansion racks

Unable to mix disk types within the enclosure, HC or HP? Exadata Smart Flash Cache Write-Back

reduces the need to mix disks

Cost Offset to some degree by gains in performance

and consolidation

54


Graphic Section Divider


Exadata Storage - Availability Best Practices Exadata Storage Availability Best Practices: Beyond Traditional Storage Ashish Ray VP of Product Management Oracle Michael Nowak Consulting

Documents