Automatic Storage Management 11g_MikeMessina_Rolta

Automatic Storage

Management (ASM)

Speaker Introduction

Michael Messina

Senior Managing Consultant Rolta, TUSC Infrastructure Services

Working with Oracle Approximately 17 years

Background includes Performance Tuning, High Availability and Disaster Recovery

Oracle Database OCP

Oracle ACE

[email protected]

www.tusc.com / www.roltasolutions.com

mailto:[email protected]

http://www.tusc.com/

http://www.roltasolutions.com/

Contents

Automatic Storage Management (ASM) Overview

ASM Structure / Components

Benefits of ASM

Any Issues?

Infrastructure Storage Options

Redundancy External, Normal, High

Multiple Paths to Storage

Database Storage

File Storage (ASM Clustered File System)

Contents

Volume Management Comparison to other Volume Managers

Use with Other Volume Managers

How ASM can replace other Logical Volume Managers

ASM in a High Availability Environment Fail Groups

Real Application Clusters (RAC) Database

DataGuard

Prepare Volumes for ASM Requirements

ASMLib for Linux

Create disks for ASM

Contents

Installing ASM Software and OS Requirements for Oracle Enterprise Linux

Installation Basics

ASM Command Line Utility What is the ASM Command Line Utility

ASMCMD Commands

Contents

Working with ASM Disk Groups Discovering Disks

Mounting/Un-mounting Disk Groups

Create ASM Disk Groups

Add Disk/Space to ASM Disk Group

Remove Disk/Space from ASM Disk Group

Change ASM Disk Group Compatibility

Rename Disk Group

Managing ASM File Access Control for Disk Groups

Drop ASM Disk Group

Contents ASM Cluster File System

Overview Oracle ACFS Driver Model

Oracle ACFS Mount Model and Namespace

Oracle ACFS Mount Registry

Oracle ACFS Snapshots

Oracle ACFS and Backup and Restore

Oracle ACFS Integration with ASM

Loading ACFS Drivers / Modules

Create/Manage ACFS File System

AUTOMATIC STORAGE MANAGEMENT (ASM)

Overview

A Volume Manager for Databases

A File System for Databases

11gR2 a File System

11gR2 a Cluster File System


ASM Instance

ASM Disk Groups

ASM Disks

ASM Files Control files

Data files, temporary data files, and data file copies

SPFILEs

Online redo logs, archive logs, and Flashback logs

RMAN backups

Disaster recovery configurations

Change tracking bitmaps

Data Pump dumpsets


January 12 / Slide 11 / Title of Document

Benefits of ASM

Dynamically Add and Remove Disk/Space while database is running

Works with Oracle Managed Files (OMF)

ASM can coexist with other storage types

ASM Can Improve I/O Performance

When in a Cluster ASM Supports Rolling Upgrades

ASM Reduces Management and Administrative Overhead

ASM can support storage for multiple Databases and Database Versions

Flexible Redundancy Options

Oracle Enterprise Manager Integration


Any Issues?

ASM Requires additional Home location for single instance implementations, meaning additional space for install is required.

For Normal or High Redundancy Disk Groups the disk for the failgroup(s) are required when the disk group is created.

RMAN is required to backup databases using ASM.

ASM does not provide any power management features, all power management features are controlled outside the ASM technology stack.


INFRASTRUCTURE


Storage Options

ASM works with virtually any storage type Direct Attached Storage

Network Attached Storage

Storage Area Networks

Solid State Disk

Storage regardless of the type is presented to the server as a RAW device(s)

Operating System on the Server Controls disk use


Redundancy

External Disk hardware controls the Redundancy

Normal 1 disk mirror or failgroup

High 2 disk mirrors or failgroups


Redundancy

Multipathing provide failover by using redundant physical paths between server

and the storage subsystem.

If one or more components fails, then applications can still access data on the storage

Multipathing used with Storage Area Network (SAN), Host Bus Adapter, interface cable, or host port on a multiported storage array.


Database Storage

Disks assigned to ASM Disk Groups

ASM Instance manages database storage to disk groups

Database datafiles, tempfiles, controlfiles, spfiles and redo logs stored in ASM Disk Groups

Database can store files in one or more Disk Groups


File Storage (ASM Clustered File System)

ASM Clustered File System (ACFS)

11gR2 added the capability for ASM to act as a file system for all operating system files with ACFS

Can act as a local filesystem or a cluster filesystem

Works natively at the OS layer to act just like any other filesystem

ASM Creation Assistant can be used to create ACFS filesystems

ASM command line utility can be used to create and manage ACFS file systems (asmcmd)


VOLUME MANAGEMENT


Comparison to other Volume Managers

ASM Requires RMAN for Database Backups while other volume managers will allow non-RMAN backups.

ASM is being taught with all Oracle Database Training and it Oracle’s Direction for the future. Other volume managers may require additional training to utilize effectively.

ASM Cluster File System adds capability to utilize ASM as a regular native OS filesystem.

ASM is a no-cost cluster file system option, 3rd party volume managers require additional license purchase and support costs.

ASM typically provides better performance

Use with Other Volume Managers

ASM co-exists with other cluster file system or native OS file systems.

Mix database and non-database files on different filesystems Use ASM for Database files and non-database files

Use OCFS2, Veritas, ext3, etc. database and non database files

Use ASM, OCFS2, Veritas, etc. for database and non-database files

Move database and non-database files between different filesystems including ASM.

How ASM can replace other Logical Volume Managers

11gR2 adds ASM Cluster File System expanding the capability of ASM to be used as a native OS filesystem. Create ASM file systems/volumes for databases

Create ACFS file systems which use ASM for storage for non-database files.

ACFS provides full volume management for virtually all file types.

ASM with ACFS however can not replace the native OS volume manager for root and boot volumes and ASM/ACFS was not designed to support root or boot volumes for the operating system but to supplement and add additional capabilities

ASM IN A HIGH AVAILABILITY ENVIRONMENT

Fail Groups

FailGroups are utilized in NORMAL and HIGH Redundancy Disk Groups.

FailGroups provide Disk Group storage protection using mirrors within Disk Groups

FailGroups can be located on multiple storage systems protecting against storage system failure

Failgroups can be located at a separate site based on network and latency conditions.

Real Application Clusters (RAC) Database

ASM was designed to provide a Highly Available Clustered Storage for Real Application Clusters

Provide highest availability for RAC External storage Redundancy using RAID 10, RAID 5, RAID 6 to

protect against individual disk failures

ASM FailGroups to protect against Storage System Failures, storage systems can be located at separate sites depending on network capability and latency.

Multiple Nodes in cluster provide protection against node failure.

DataGuard

Data Guard Provides for long distance cross site failover

Uses ASM effectively

ASM can simplify the building and maintaining Data Guard Standby database

RAC capable standbys from RAC primary or single instance standby from RAC primary.

Closes the maximum availability and scalability gap that RAC and ASM alone can not provide.

PREPARE VOLUMES FOR ASM

Requirements

For Linux ASMLib utility is not required, however recommended and considered best practice. ** Not support in Red Hat 6 for ASMLib going forward based on most recent information

For Linux ASMLib should be installed and configured

RAW volumes presented to the host/operating System

RAW volumes partitioned

Permissions and ownership of volumes for oracle and dba or ASM Admin group

Make permissions and ownership of volumes persistent across host restarts

ASMLib for Linux

Check Installation rpm –qa | grep oracleasm

Install ASMLib http://www.oracle.com/technology/software/tech/linux/asmlib/rhel5.html

rpm -i oracleasm-support-2.1.3-1.el5.i386.rpm

rpm -i oracleasm-2.6.18-128.1.6.el5-2.0.5-1.el5.i686.rpm

rpm -i oracleasmlib-2.0.4-1.el5.i386.rpm

rpm -i oracleasm-2.6.18-128.1.6.el5debug-2.0.5-1.el5.i686.rpm

rpm -i oracleasm-2.6.18-128.1.6.el5-debuginfo-2.0.5-1.el5.i686.rpm

rpm -i oracleasm-2.6.18-128.1.6.el5PAE-2.0.5-1.el5.i686.rpm

rpm -i oracleasm-2.6.18-128.1.6.el5xen-2.0.5-1.el5.i686.rpm

Configure ASMLib /etc/init.d/oracleasm configure

http://www.oracle.com/technology/software/tech/linux/asmlib/rhel5.html

Create disks for ASM

Partition RAW volumes to be used for ASM fdisk /dev/sda

Change Group at min and permissions on volume and partition chmod 660 /dev/sda

chmod 660 /dev/sda1

chown oracle:dba /dev/sda

chown oracle:dba /dev/sda1

Place Group and permission changes in /etc/rc.d/rc.local

Initialize partitions on devices into ASM Disks /etc/init.d/oracleasm createdisk ASMDSK1 /dev/sda1

INSTALLING ASM

Software and OS Requirements for Oracle Enterprise Linux

OS Requirements Enterprise Linux At least 1GB of free available memory on server

At least 1.8GB of free swap

Oracle user

Oracle dba group

Ensure the following exist in the /etc/security/limits.conf * hard nproc 16384

* hard nofile 65536

Ensure that the limits are configured to set for new sessions by ensuring the following are in the /etc/pam.d/system-auth file. session required pam_limits.so

session required pam_unix.so



Ensure that the following are included in the .bash_profile for the oracle user ulimit -n 65536

ulimit -u 16384

Ensure oracle primary group dba $ usermod –g dba oracle

set the following in the /etc/sysctl.conf kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 4294967295

kernel.shmall = 268435456

kernel.sem=250 32000 100 128

net.ipv4.ip_local_port_range = 1024 65000 < 11gR1

net.ipv4.ip_local_port_range = 9000 65500 >= 11gR2

net.core.rmem_default=4194304

net.core.wmem_default=262144

net.core.rmem_max=4194304

net.core.wmem_max=1048576

fs.file-max = 6815744

fs.aio-max-nr = 1048576



CD3 rpm -Uvh libaio-0 rpm -Uvh libaio-devel-0.3.105-2.i386.rpm

CD4 rpm -Uvh elfutils-libelf-0 rpm -Uvh elfutils-libelf-devel-0.91.1-3.i386.rpm

CD1 rpm -Uvh unixODBC-2.2.11-1.RHEL4.1.i386.rpm


Installation Basics

Ensure OS compatibility for the ASM version to be installed

Ensure all OS pre-reqs are satisfied

For ASM the latest ASM version is recommended to be used even for lower versions of the database software

Obtain the latest ASM Software currently 11.2 and is included in the Grid Infrastructure Software Installation

The Oracle Universal Installer greatly simplifies the installation process for the Database and Grid Infrastructure software.

ASM COMMAND LINE UTILITY

What is the ASM Command Line Utility

Command line interface for management of ASM disk groups and ACFS volumes

Run from Grid Infrastructure home

<GRIDHOME>/bin/asmcmd

ASMCMD Commands

Instance Management iostat Displays statistics for mounted devices.

lsct Lists information about current ASM clients.

lsop Lists the current operations on a disk group, database, or ASM instance.

lspwusr List the users from an Oracle PWFILE file.

orapwusr Add, drop, or change an Oracle PWFILE user.

shutdown Shuts down an ASM instance.

spcopy Copies a SPFILE.

startup Starts up an ASM instance.

ASMCMD Commands

File Access chgrp Changes the group of a file.

chmod Changes permissions of a file.

chown Changes the owner of a file.

groups List the user groups that a user belongs to.

grpmod Adds or removes users from an existing user group.

lsgrp Lists ASM access control list (ACL) groups.

lsusr Lists users in a disk group.

mkgrp Creates a new user group.

mkusr Adds a user to disk group.

passwd Changes the password of a user in a disk group.

rmgrp Deletes a user group.

rmusr Deletes a user from a disk group.

ASMCMD Commands

File Management cd Changes the current directory to the specified directory.

du Displays the total disk space occupied by ASM files in the specified ASM directory and all of its subdirectories, recursively.

find Lists the paths of all occurrences of the specified name (with wildcards) under the specified directory.

ls Lists the contents of an ASM directory, the attributes of the specified file, or the names and attributes of all disk groups.

lsof Lists the open files.

mkalias Creates an alias for system-generated filenames.

mkdir Creates ASM directories.

pwd Displays the path of the current ASM directory.

rm Deletes the specified ASM files or directories.

rmalias Deletes the specified alias, retaining the file alias points to.

ASMCMD Commands

Disk Group Management chdg Changes a disk group (add, drop, or rebalance).

chkdg Checks or repairs a disk or failure group.

cp Enables you to copy files between ASM disk groups on a local instance and remote instances.

dgdrop Drops a disk group.

lsattr Lists the attribute and attribute settings of a disk group.

lsdg Lists disk groups and their information.

lsdsk Lists disks visible to ASM.

lsgrp Lists ASM access control list (ACL) groups.

md_backup Creates a backup of all of the mounted disk groups.

md_restore Restores disk groups from a backup.

mkdg Creates a disk group.

mount Mounts a disk group.

ASMCMD Commands

Disk Group Management (cont.) offline Offline a disk or a failure group that belongs to a disk

group.

online Onlines a disk or a failure group that belongs to a disk group.

rebal Rebalances a disk group.

remap Repairs a range of physical blocks on a disk.

setattr Sets attributes in an ASM disk group.

umount Dismounts a disk group.

ASMCMD Commands

ACFS – ASM Clustered File System volcreate Create a volume

voldelete Delete a volume

voldisable Disable a volume.

volenable Enable a Volume

volinfo Volume Information,

volresize Resize a volume

volset Change existing attribute

volstat Report volume I/O statistics.

WORKING WITH ASM DISK GROUPS


Discovering Disks

You can discover disk groups using ASM lib or the ASM instance. ASMLib

/etc/init.d/oracleasm scandisks

/etc/init.d/oracleasm listdisks

ASM Instance SQL> select name, path from v$asm_disk ;


Set ORACLE_HOME to Grid Infrastructure installation location

SET ORACLE_SID to ASM instance name typically +ASM

A disk group can be created using SQL Plus or the ASM command line utility.


External Redundancy SQL Plus

SQL> create diskgroup UNDO external redundancy disk 'ORCL:UNDO1' ;

asmcmd Create diskgroup_config.xml

• <dg name="dgroup1" redundancy=“external">

• <dsk string="/dev/raw/disk1" />

• <a name="compatible.asm" value="11.2"/>

• <a name="compatible.rdbms" value="11.2"/>

• </dg>

ASMCMD> mkdg diskgroup_config.xml


Normal Redundancy SQL Plus

SQL> create diskgroup test_failgroup normal redundancy

FAILGROUP fg1 DISK ‘ORCL:ASMDSK1' NAME ASMDSK1

FAILGROUP fg2 DISK ‘ORCL:ASMDSK2' NAME ASMDSK2 ;


• <dg name="dgroup1" redundancy="normal">

• <fg name="fg1">



• </fg>

• <fg name="fg2">



• </fg>



• </dg>



High Redundancy SQL Plus

SQL> create diskgroup test_failgroup high redundancy

FAILGROUP fg1 DISK '/dev/oracleasm/disks/ASMDSK1' NAME ASMDSK1

FAILGROUP fg2 DISK '/dev/oracleasm/disks/ASMDSK2' NAME ASMDSK2

FAILGROUP fg3 DISK '/dev/oracleasm/disks/ASMDSK2' NAME ASMDSK3 ;



• <dg name="dgroup1" redundancy=“high">

• <fg name="fg1">



• </fg>

• <fg name="fg2">



• </fg>

• <fg name="fg3">



• </fg>



• </dg>


Mounting/Un-mounting Disk Groups

Mounting and Un-mounting Disk Groups can be done via SQL Plus or the asmcmd utility Mount

SQL Plus • SQL> alter diskgroup undo mount ;

asmcmd • asmcmd> mount undo ;

Un-mount SQL Plus

• SQL> alter diskgroup undo dismount ;

asmcmd • asmcmd> umount undo ;

Add Disk/Space to ASM Disk Group

Space is added to a disk group by the addition of ASM disks/volumes. SQL Plus

SQL> alter diskgroup test add disk ‘ORCL:ASMDSK5‘

Remove Disk/Space from ASM Disk Group

Space is removed from an ASM Disk Group by removing disks from the diskgroup. SQL Plus

SQL> alter diskgroup test_failgroup drop disk ASMDSK2 ;

Change ASM Disk Group Compatibility

The Disk Group Compatibility can be change after the disk group is created provided it does not exceed compatibility in the environment being used. The COMPATIBLE.ASM and COMPATIBLE.RDBMS dictate the instance and RDBMS capability instance parameter settings.

COMPATIBLE.ASM COMPATIBLE.RDBMS ASM Instance RDBMS Instance

10.1 10.1 >=10.1 >=10.1

11.1 10.1 >=11.1 >=10.1

11.2 11.1 >=11.2 >=11.1

11.2 11.2 >=11.2 >=11.2

Disk Group can only be changed upward (so be careful) SQL Plus

SQL> ALTER DISKGROUP data SET ATTRIBUTE 'compatible.asm' = '11.1';

Rename Disk Group

Starting will 11g Release 2 a disk group can be renamed using the renamedg command line utility asmcmd

renamedg dgname=olddiskgroup newdgname=newdiskgroup

** Note: If Disk Group Contains Database Files the database files will need to be relocated. This will require off lining the tablespaces that have datafiles on the diskgroup or shutting the database down prior to renaming the diskgroup. Then once diskgroup is rename relocating all files with alter database rename command for new Disk Group Name.


Managing file and directory is done using the asmcmd command line utility

Managing access in asm is much like managing file and directory access in unix operating systems.

You have users, groups, directory and file permissions controlled through the use of commands mkusr

mkgrp

rmgrp

rmusr

chown

chgrp

chmod


mkusr – Create New User asmcmd> mkusr dg1 myuser ;

mkgrp – Create New Group asmcmd> mkgrp dg1 mygroup ;

grpmod – Add/Remove Users for Group -a Add User -d Remove User

grpmod –a dg1 mygroup myuser

grpmod –d dg1 mygroup myuser

rmgrp – Remove Group asmcmd> rmgrp dg1 mygroup ;

rmusr – Remove User asmcmd> rmusr dg1 myuser ;


chown ASMCMD> chown db1 +data/orcl/datafile/*

ASMCMD> chown –R db1 +data/orcl

ASMCMD> chown db1:backup_user +data/controlfile.f

chgrp asmcmd> chgrp backup_users +data/controlfile.f ;

asmcmd> chgrp –R backup_users +data/* ;

chmod ASMCMD> chmod 644 +data/controlfile.f

ASMCMD> chmod go+rw +data/controlfile.f

ASMCMD> chmod ug-w +data/orcl/datafile/*

ASMCMD> chmod –R 600 +data/orcl

Drop ASM Disk Group

A disk group can be dropped using either SQL Plus or the asmcmd command line utility SQL Plus

DROP DISKGROUP dg1 ;

** If Disk Group has files then use the INCLUDING CONTENTS clause

SQL> DROP DISKGROUP disk_group_1 INCLUDING CONTENTS;

asmcmd asmcmd> dgdrop dg1 ;

** If Disk Group has files then use the –r option

asmcmd> dgdrop –r dg1 ;

ASM CLUSTER FILE SYSTEM

Overview

Requires installation of 11g Release 2 (11.2) Grid Infrastructure Software

ACFS executes as a UNIX (POSIX and X/OPEN compatible) file system for Linux and UNIX and ACFS works as a Windows Filesystem on Windows.

Applications and OS commands work natively against ACFS

Exabyte capable files and file system capacities on 64 bit platforms

Peer to peer, multi-node shared filesystem with direct, coherent and cache I/O to ASM Storage.

Works as a Clustered or non-clustered filesystem

Oracle ACFS are provided with “single node file system coherency” for single host or when distributed across cluster members.

Oracle ACFS file system can be used for an Oracle Database Home file system.

Oracle ACFS Driver Model

Oracle ACFS is installed as a dynamically loadable Operating System filesystem driver.

Driver is implemented as a Virtual File System (VFS)

Integrated with the native operating system

Oracle ACFS Mount Model and Namespace

Hierarchical filesystem Files

Sub-directories

Tree-structured Namespace Files are the leaf nodes

Single-filesystem naming model

Not designed to be a root filesystem or bootable.

Oracle ACFS Mount Registry

Multi-platform mount registration facility

Automatically mount file systems on all nodes in a cluster that are registered

Clustered and non-clustered filesystems are registered in the Mount Registry, most useful for clusters really Clustered see acfsutil command

Non-clustered see srvctl add filesystem command

Creates the mount point if it does not exist

Registry mount actions will automatically mount associated ASM disk groups

Oracle ACFS Snapshots

Point in time copy of an ACFS filesystem

Works very similar to SAN Volume or Flash Copies

Snapshots are immediately available for use after creation

Always online while the file system is mounted

Snapshot storage is maintained within the file system

Snapshots are administered with the acfsutil snap commands

63 snapshot views supported for each file system

Oracle ACFS and Backup and Restore

Uses same backup capabilities as other filesystems for the host operating system environment.

ACFS snapshots can be dynamically created and used to present a consistent view of an active file system for a backup

NOTE: Windows backup applications that depend upon reparse points or the Windows Volume Shadow Copy Service (VSS) may not be supported.

Oracle ACFS Integration with ASM

ACFS requires ASM for storage

Always integrated with ASM

Configured with a traditional device file

ACFS filesystem inherits ASM storage management features like balanced distribution, mirroring, dynamic resizing, etc.

Driver established connection with ASM instance

Always unmount ACFS filesystems prior to ASM instance shutdown or prior t0 unmounting a disk group to avoid the filesystem being put into an error state.

Loading ACFS Drivers / Modules

To utilize ACFS the ACFS drivers and modules must be loaded into the operating system For Grid Infrastructure installations in a cluster the drivers and

modules are loaded automatically

Single instance Grid Infrastructure installations the ACFS Drivers/Modules must be loaded Manaually

Load ACFS Drivers/Modules Manually Log into the host operating system as root

<GRIDHOME>/bin/acfsload start

Place the load command into /etc/rc.d/rc.local for the load to be persistent across node restarts


Must first have a Disk Group for the ACFS file system to use

Disk Group Compatibility must be 11.2.0 or higher SQL> select group_number, name, compatibility,

database_compatibility from v$asm_diskgroup;

alter diskgroup avm SET ATTRIBUTE 'compatible.advm' = '11.2.0.0'

Create an ACFS File System through ASMCA or asmcmd Create using asmcmd – ASM command line utility

volcreate -d diskgroupa -s 10G volume

Get ACFS volume information volinfo –a dg1

Enable volumes in a Disk Group All - volenable –a dg1

Volume – volenable –d dg1 <volumename>


Resize a ACFS volume in a Disk Group volresize -d <diskgroupname> -s 2G [ -f ] <volumename>

ACFS Volume Statistics volstat -d <diskgroup_name> <volume_name>

Disable ACFS volume voldisable -d <diskgroup_name> <volume_name>

Remove an ACFS volume voldelete -d <diskgroup_name > <volume_name>

Thank You

Thank you!


Automatic Storage Management 11g_MikeMessina_Rolta

Documents

asm contents

cluster asm

asm software

storage types asm

asm requirements asmlib

administrative overhead

rolling upgrades asm

asm technology stack