DOUG – Technology Day - DBCloudShifu€¦ · DOUG – Technology Day ... Start of 'ora.crsd' on ’racnode2' succeeded ... $ crsctl stat res -t ...

@ViscosityNA

1_Title Slide

DOUG – Technology Day

My Top Ten 12c ASM Questions!Nitin Vengurlekar!

www.viscosityna.com 1

Nitin Vengurlekar

•  18 Years with Oracle  6 years with Oracle Support  9 years with RAC Product Management  3 years a “Private Database Cloud” Evangelist

•  Currently serving as CTO at Viscosity –  Work with numerous customers on consolidation, rationalization

planning. –  Work with customers on Performance Tuning, Systems and Database

Migration •  Follow me on Twitter: dbcloudshifu


<Insert Picture Here>

Agenda

• ASM Overview – 2 minute tour• Top Ten Questions

– ASM Architecture– ASM Configuration & Performance– ACFS

ASM Architecture - Review

•  Automatic Storage Management (ASM) instance–  Instance that manages the diskgroup metadata

•  Disk Groups – Logical grouping of disks– Determines file mirroring options

•  ASM Disks– LUNs presented to ASM

•  ASM Files– Files that are stored in ASM disk groups are called ASM files, this includes

database files

ASM Legacy Configuration

Disk1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7

Disk Group A Disk Group B

Database Database Database

File 1 File 2

File 3 File 4

ASM Cluster Pool of Storage

OracleRACServers

ASM Instance

ASM Instance

ASM Instance

ASM Instance

ASM Instance

Database Instance

Database Instance

Database Instance

Database Instance

Database Instance

1-‐1 ASM to Server

Shared Disk Groups

Wide File Striping

Flex ASM Configuration

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7



File 1 File 2

File 3 File 4


Oracle RAC Servers

ASM Instance Database Instance Database Instance Database Instance Database Instance Database Instance

ASM Instance ASM Instance Database Instances connect to any ASM Instance

3 ASM Instances

Shared Disk Groups

Wide File Striping


Top 10 ASM Questions ASM Architecture

Top 10 ASM Questions

Q. What init.ora parameters does a user need to configure for 12c ASM instances?

A.  There are virtually no explicit settings needed for 12c ASM. The default parameter settings work perfectly well. Required parameters are inserted/updated by crsd asm agentsHowever, for large database consolidation need to follow this

guideline for ASM Processes: Non-Exadata environment: For n >= 10, PROCESSES = 10*n + 450 For Exadata environment = MAX(450 + 10*n, 1024)

Top 10 ASM QuestionsQ. How does the database interact with the ASM instance and

how do I make ASM go faster?

A.  ASM is not in the I/O path so ASM does not impede the database file access. Since the RDBMS instance is performing raw I/O, the I/O is as fast as possible.

RDBMS and local ASM Instance Interaction

Server

Operating System

DATABASE ASM

(1) Database opens file

(1A) OPEN

(1B) Extent Map

(2) Database Reads file

(2A) READ

(2B) I/O Completes(3) Database Creates file

(3A) CREATE

(3B) Allocates file

(3C) Extent Map

(3D) Initializes file (3D) Commit

Top 10 ASM Questions Q. What is the best LUN size for ASM

A.  There is no best size! In most cases the storage team will dictate based on their standardized LUN size. The ASM admin merely has to communicate the ASM Best Practices and application characteristics to storage folks :

•  Need equally sized / performance LUNs •  Minimum of 4 LUNs•  The capacity requirement•  The workload characteristic (random r/w, sequential r/w) & any response time

SLA Using this info, and their standards, the storage folks should build a nice LUN group

set for you.


Top 10 ASM QuestionsFlex Cluster

Q. What is the difference between Flex ASM and Flex Cluster?

A. Flex Cluster consists of a cluster with Hub and leaf node configuration only. •  Hub nodes have direct access to storage (ocr/voting disk, etc).•  Leaf nodes do not have direct access to shared storage.•  Flex Cluster requires Flex ASM. Flex ASM does not require a Flex

Cluster.•  In Flex cluster node, the Flex ASM can run on the Hub nodes

servicing clients across the Hub nodes of the Flex cluster.

www.viscosityna.com

Image courtesy of Toad

Top 10 ASM QuestionsFlex ASM

Q. But then what is Flex ASM and when would I use it?

A.  Flex ASM is a flexible architecture for ASM deployment. No more hard-wiring ASM instances to nodes. Databases are “load balanced“ connected to appropriate ASM instances

Dependent on three sub featueares – ASM Listener, ASM Network, and Shared pasword file

Flex ASM Configuration

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7



File 1 File 2

File 3 File 4


Oracle RAC Servers

ASM Instance Database Instance Database Instance Database Instance Database Instance Database Instance

ASM Instance ASM Instance Database Instances connect to any ASM Instance

3 ASM Instances

Shared Disk Groups

Wide File Striping

• Three deployment models for ASM Clustering•  FlexASM – Pure 12c Mode

•  Standard Mode Configuration (aka Mixed Mode)• Supports pre-12c and 12c Databases• Set asm cardinality=‘all’


Using ASMCA

$ ./asmca –silent –convertToFlexASM –asmNetworks eth1/192.168.0.1 –asmListenerPort 1521 CRS-2673: Attempting to stop 'ora.crsd' on ’racnode1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode1' [...] CRS-2672: Attempting to start 'ora.crsd' on 'racnode1' CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded Cluster bounced in node racnode1 CRS-2673: Attempting to stop 'ora.crsd' on 'racnode2' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode2' [...] CRS-2672: Attempting to start 'ora.crsd' on 'racnode2' CRS-2676: Start of 'ora.crsd' on ’racnode2' succeeded Cluster bounced in node racnode2



Q. What’s with this ASM listener and do I really need a listener for ASM…seriously!!?

A. •  New global CRS resource to support Flex ASM•  Runs where ASM instance is running

•  ASM instances registers with all ASM listeners•  Functionally similar to SCAN Listener

•  Clients [DB instances] connect to ASM using ASM listener endpoints•  Clients connect using connect data credentials defined by CSS GMS•  Clients try best connection (local) first, if no local-node ASM, then connect to

(any) remote ASM instance in the cluster•  Connectivity is load balanced across ASM instances

Flex ASM - ASM Listener

Flex ASM - ASM Network

• In previous versions Oracle Clusterware required:• A public network for client application access• One or more private networks to carry Cache Fusion and ASM traffic within the cluster

• Flex ASM adds option for ASM network, • Used for communication between ASM and its clients• Extent map shipping across this network• OCR metadata bursty traffic on failover or resource state changes • SQL*Net credential handling for clients• Can continue to share the CSS private network


ASM 12c Implementation

We are running a 4 node RAC cluster with ASM default cardinality [oracle@rac01 trace]$ crsctl get node role status -all Node 'rac01' active role is 'hub' Node 'rac03' active role is 'hub' Node 'rac02' active role is 'hub' Node 'rac04' active role is 'hub’

ASM Config [oracle@rac02 ~]$ srvctl config asm -detail ASM home: /u01/app/12.1.0/grid Password file: +CRSDATA/orapwASM ASM listener: LISTENER ASM is enabled. ASM instance count: 3 Cluster ASM listener: ASMNET1LSNR_ASM

First let's check we are really running in FlexASM mode: oracle@rac02 ~]$ asmcmd showclustermode ASM cluster : Flex mode enabled

Running 2 node Serverpool [oracle@rac02 ~]$ srvctl status serverpool -serverpool naboo Server pool name: naboo Active servers count: 2

ACFS is also enabled for storing trace data [oracle@rac02 ~]$ srvctl status filesystem ACFS file system /u02/app/oracle/acfsmounts is mounted on nodes rac01,rac02,rac03 srvctl status asm -proxy ADVM proxy is running on node rac01,rac02,rac03,rac04


[oracle@rac01 ~]$ crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE rac01 STABLE ONLINE ONLINE rac02 STABLE ONLINE ONLINE rac03 STABLE ONLINE ONLINE rac04 STABLE

You notice that we have 4 ASM listeners one on each node in the Cluster. You'll see the process as the following on each node: [oracle@rac01 ~]$ ps -ef |grep -i asmnet ooracle 6646 1 0 12:19 ? 00:00:00 /u01/app/12.1.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit


The pdbdata, crsdata and fra disk group resources are started on all nodes except node 4 ora.CRSDATA.dg ONLINE ONLINE rac01 STABLE ONLINE ONLINE rac02 STABLE ONLINE ONLINE rac03 STABLE OFFLINE OFFLINE rac04 STABLE ora.FRA.dg ONLINE ONLINE rac01 STABLE ONLINE ONLINE rac02 STABLE ONLINE ONLINE rac03 STABLE OFFLINE OFFLINE rac04 STABLE ora.PDBDATA.dg ONLINE ONLINE rac01 STABLE ONLINE ONLINE rac02 STABLE ONLINE ONLINE rac03 STABLE OFFLINE OFFLINE rac04 STABLE


Since we have a two node Serverpool we have 2 DB instances connecting to 2 of 3 ASM instances…maybe not too not exciting ….since it could be all local connections #crsctl stat res –t ora.tatooine.db 1 ONLINE ONLINE rac03 Open,STABLE 2 ONLINE ONLINE rac01 Open,STABLE --------------------------------------------------------------------------------


srvctl modify srvpool -serverpool naboo -min 1 -max 4 Crsctl stat res -t ora.tatooine.db 1 ONLINE ONLINE rac03 Open,STABLE 2 ONLINE ONLINE rac01 Open,STABLE 3 ONLINE ONLINE rac02 Open,STABLE 4 ONLINE ONLINE rac04 Open,STABLE ora.asm 1 ONLINE ONLINE rac03 STABLE 2 ONLINE ONLINE rac01 STABLE 3 ONLINE ONLINE rac02 STABLE [oracle@rac03 ~]$ srvctl status asm -detail ASM is running on rac01,rac02,rac03 ASM is enabled.


Database from node4 (where ASM is not running), connects to ASM1 on node1. So, what the does the Database have to say about ASM connectivity:

NOTE: ASMB registering with ASM instance as client 0x10001 (reg:1377584805) NOTE: ASMB connected to ASM instance +ASM1 (Flex mode; client id 0x10001) NOTE: ASMB rebuilding ASM server state NOTE: ASMB rebuilt 2 (of 2) groups SUCCESS: ASMB reconnected & completed ASM server state



Q. How to relocate Database client to different ASM?

A. Clients can be relocated (shifted) to an available ASM if ASM is gracefully shutdown or if ASM instance fails. The connection is load balanced to an available instance.

Clients can be manually relocated using the following:

SQL> ALTER SYSTEM RELOCATE CLIENT '<instance_name>:<db_name>'; Query GV$ASM_CLIENT to determine instance_name and db_name.

[oracle@rac03 ~]$ asmcmd lsct -g pdbdata Instance_ID DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group 1 +ASM CONNECTED 12.1.0.2.0 12.1.0.2.0 +ASM1 PDBDATA 2 +ASM CONNECTED 12.1.0.2.0 12.1.0.2.0 +ASM2 PDBDATA 3 +ASM CONNECTED 12.1.0.2.0 12.1.0.2.0 +ASM3 PDBDATA 2 tatooine CONNECTED 12.1.0.2.0 12.1.0.0.0 tatooine_1 PDBDATA 1 tatooine CONNECTED 12.1.0.2.0 12.1.0.0.0 tatooine_2 PDBDATA 3 tatooine CONNECTED 12.1.0.2.0 12.1.0.0.0 tatooine_3 PDBDATA 3 tatooine CONNECTED 12.1.0.2.0 12.1.0.0.0 tatooine_4 PDBDATA


Let’s say we wanted to move a database instance from one ASM instance connection to another ASM for planned outage (patching Clusterware) SQL> alter system relocate client 'tatooine_1:tatooine'; NOTE: ASMB registering with ASM instance as client 0x10002 (reg:2531435813) NOTE: ASMB connected to ASM instance +ASM2 (Flex mode; client id 0x10002) NOTE: ASMB rebuilding ASM server state NOTE: ASMB rebuilt 2 (of 2) groups NOTE: ASMB rebuilt 19 (of 19) allocated files NOTE: fetching new locked extents from server NOTE: 0 locks established; 0 pending writes sent to server SUCCESS: ASMB reconnected & completed ASM server state Fri Jan 10 21:09:24 2014 NOTE: ASMB relocating from ASM instance +ASM2 (ASM-initiated) NOTE: ASMB registering with ASM instance as client 0x10002 (reg:3934509467) NOTE: ASMB connected to ASM instance +ASM3 (Flex mode; client id 0x10002) NOTE: ASMB rebuilding ASM server state NOTE: ASMB rebuilt 2 (of 2) groups NOTE: ASMB rebuilt 19 (of 19) allocated files NOTE: fetching new locked extents from server NOTE: 0 locks established; 0 pending writes sent to server SUCCESS: ASMB reconnected & completed ASM server state


[oracle@rac02 ~]$ srvctl status instance -d tatooine -i tatooine_1 Instance tatooine_1 is running on node rac03 [oracle@rac02 ~]$ srvctl status instance -d tatooine -i tatooine_2 Instance tatooine_2 is running on node rac01 [oracle@rac02 ~]$ srvctl status instance -d tatooine -i tatooine_3 Instance tatooine_3 is running on node rac02 [oracle@rac02 ~]$ srvctl status instance -d tatooine -i tatooine_4 Instance tatooine_4 is running on node rac04


Top 10 ASM QuestionsQ. What new disk management functions were introduced in 12c

A.  There are several key disk management features, and some specific to Exadata

• Fast Disk Resync & Checkpoints

• Fast Disk Replacement

• ASM Disk Scrubbing

• Rebalance Optimizations

ASM New Features Disk Management Optimizations – Scalable Resync

•  In Legacy ASM Disk Online operation was single-threaded•  This usually does not work well when the disk has been offline for a long time or when

multiples disks have gone offline

•  12c Fast Disk Resync Feature•  Allows Admins to throttle the resync operation•  Power limit can be set for disk resync operations from 1 to 1024

SQL> alter diskgroup reco online disk reco_0004 power 16

ASMCMD> online –G reco –D reco_004 –power 16

•  Resync operation now has a time estimate (EST_MINUTES) in V$ASM_OPERATION

•  Relocate and rebalance code is merged

ASM New Features Disk Management Optimizations – Disk Resync Checkpoints

•  In Legacy ASM if Disk Resync operation was interrupted it had to be re-started from the beginning

•  12c ASM Disk resync incorporates disk online “Check Points”•  Interrupted resync operations are automatically restarted

•  Previously completed phases of the resync are skipped

•  Resync processing restarts at the beginning of the first remaining incomplete phase •  Every 1000 extents are “batch” cleared in SR

ASM New Features Disk Management Optimizations – Fast Disk Replacement

•  In Legacy ASM, if a disk “truly” failed, it needed to be replaced and re-added to diskgroup•  This required an expensive re-partnership and rebalance

•  12c ASM allows fast, low-overhead replacement of failed disks•  Replace new disk in same slot as failed disk SQL>alter diskgroup replace disk rec_0004 with ‘/dev/mapper/mpathe’

•  Replacement disk takes the same name as the original disk and becomes part of the same failure group

•  Replacement disk is re-populated with copies of ASM extents from other disks

•  Disk group rebalance is not needed

ASM New Features Disk Management Optimizations - Failure Group Repair timer

•  ASM 12c allows Failure Group repair time

•  Useful for protecting against transient failgroup issues, such as controller issues or poor connectivity in Stretch Clusters

•  New disk group attribute, failgroup_repair_time•  Similar to existing disk repair time•  Default setting is 24 hours

SQL> alter diskgroup reco set attribute ‘failgroup_repair_time = 36h’;

ASM New Features Disk Management Optimizations - Even Read

•  Historically, ASM always reads the primary copy of mirrored data•  For stretch clusters preferred read failure groups could be configured

•  ASM 12c “Even Read” distributes data reads evenly across all disks•  Enabled by default•  Each read request is sent to the least loaded available disk

•  This is an attempt to equalize disk reads across each disk group•  Does not measure the latency or performance of each read.

•  Even Read does not replace the functionality provided by preferred read failure groups -> Preferred read failure groups can still be configured

•  V$ASM_DISKGROUP reflects “even read” status and “even read” IOs shown in V$ASM_DISK_IOSTAT

Disk Management Optimizations – Data Scrubbing

• The database checks for logical consistency on reading data• Currently only if a logical corruption is detected can automatic recovery can be performed using the mirror copies• For seldom accessed data, corrupted data could be present in the system for a long time between reads• Possibility that all mirrors are corrupted increases over time

• ASM 12c Scrubbing features validates data integrity:• New background [master] process SCRB coordinates SCRN cycle for • Can be offloaded to Cells in Exadata• Auto-magic error correction using mirrored data

ASM New Features

Disk Management Optimizations – Data Scrubbing

• ASM 12c scrubbing occurs in two ways:• As part of rebalance operations

• Requires content.check attribute set to enable SQL> alter diskgroup data set attribute ‘content.check’ = ‘TRUE’

• On demand scrubbing• Scrubbing can be performed on a Disk Group, on individual files or individual disks SQL> alter diskgroup data scrub repair; SQL> alter diskgroup data scrub disk data_0004 norepair

power high;

ASM New Features

Disk Management Optimizations –Rebalance Improvements

•  Establish upfront estimates before actually doing a rebalance operation

EXPLAIN WORK FOR ALTER DISKGROUP pdbdata DROP DISK pdbdata_0008; Explained. SELECT est_work FROM V$ASM_ESTIMATE; EST_WORK -------- 2244

•  Improved accuracy of Rebalance estimates •  Phase of Rebalance now exposed in v$ASM_OPERATION

•  Code optimization of lock acquisition/release significantly improves rebalance overall elapsed time

ASM New Features

Engineered Systems Optimizations - Rebalance Offload

• Most extent relocations performed by a rebalance operation offloaded to Exadata Storage Server.

• A single offload request can replace multiple read and write I/O requests.

• Offloading relocations avoids sending data to the ASM host, consequently improving rebalance performance

• Rebalance uses fewer Compute server resources, I/O resources and executes quicker

ASM New Features


Top 10 ASM QuestionsACFS

44

3rd Party FSApplication

Extending ASM to Manage ALL Data

Automatic Storage Management (ASM)

ASM Instance Managing Oracle DB Files

ASM Cluster & Single Node File System (ACFS)

Database RAC

ACFS Snapshot

ASM Files

Dynamic Volume Manager

Dynamic Volumes

ACFS FS Oracle Bin

3rd Party FS

DB DatafilesDB DatafilesDB Datafiles

OCR & Voting files

ASM

Disk

Group

45

•  General purpose scalable file system•  Journaling, extent based•  Single node and cluster•  POSIX, X/OPEN file system solution for UNIX/Linux•  Windows file system solution for Windows platforms

•  Accessible through NAS protocols (NFS, CIFS)•  Leverages ASM technology•  Integrated with Oracle Clusterware for cluster support•  Multi OS platform Integrated with Oracle system mgt tools

•  Oracle installation and configuration•  Enterprise Manager and ASM Storage mgt tools•  Native OS File System Management tools

What is ASM Cluster File System (ACFS)

46

ACFS Features • Provides filesystem snapshots (FCOW) • File system integrity and fast recovery via ACFS metadata checksums

and journaling. • ACFS designed as a peer to peer, multi-node, shared file system

model and delivers coherent data access • ACFS file system is installed as a dynamically loadable OS VFS driver

Top 10 ASM QuestionsQ. Are database files supported on ACFS?

A.  Yes. In 12c, fully supported. In fact, on ODA, ACFS is the default configuration for database files

Top 10 ASM QuestionsQ. Is ACFS supported on Exadata ?

A.  Yes. In fact, with12c GI stack, all snapshot features are supported; however, ACFS Encrpytion, replication, and tagging are not supported


Q. What is ASM [ADVM] Proxy Instance? A. ASM proxy instance is configured to support the ACFS/ADVM. The ASM proxy instance can run on the same node as ASM instance or on a different node in a flex ASM. [oracle@racnode1 ~]$ ps -ef|grep pmon oracle 9098 1 0 17:50 ? 00:00:00 asm_pmon_+ASM1 oracle 9423 1 0 17:50 ? 00:00:00 ora_pmon_tatooine_1 oracle 10122 1 0 17:51 ? 00:00:00 apx_pmon_+APX1 oracle 16897 15751 0 18:42 ttyS0 00:00:00 grep pmon


Diagram shows ACFS/ADVM on node 2 connects to local ASM proxy, which connects to ASM instance on remote node

ACFS New Features HANFS

• High Availability NFS (HANFS) provides uninterrupted NFS services:•  Exported file systems are exposed using Highly Available Virtual IPs (HAVIPs)

•  Oracle Clusterware manages the NFS exports and HAVIPs•  Services are automatically migrated if the current node fails

HAVIP HAVIP

NFS Export

NFS Export

Top 10 ASM Questions - Summary –

•  Lots of new features test and try•  I didn’t even talk about AFD J•  ASM and ASM features will be embedded for many new

and upcoming features •  Rapid Home Provisioning•  ODA

/company/viscosity-north-america!

@ViscosityNA!

www.viscosityna.com/blog/!

Midway Office Tower!13612 Midway Road!Dallas TX 75244!

Follow Us!


More detailed information is available at viscosityna.com or by talking to a real

person at 469.444.1380

54

DOUG – Technology Day - DBCloudShifu€¦ · DOUG – Technology Day ... Start of 'ora.crsd' on ’racnode2' succeeded ... $ crsctl stat res -t ...

Documents