IBM Spectrum Scale for File and Object Storage

© Copyright IBM Corporation 2015. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

cIS4107IBM Spectrum Scale for File and Object storage

� Tony Pearson

� Master Inventor and Senior IT Specialist

� IBM Corporation

2IBM Systems Technical University, October 5-9 | Hilton Orlando


Abstract

IBM is ranked #1 in Software Defined Storage, and the latest release of IBM Spectrum Scale now offers additional support for connecting files and objects with the rest of your data center. Come learn how Spectrum Scale and Elastic Storage Server can help you.



This week with Tony Pearson

Day Time TopicMonday 10:15am Opening Session – Storage

01:45pm IBM's Cloud Storage Options

Tuesday 11:30am Software Defined Storage -- Why? What? How?

03:15pm The Pendulum Swings Back –Understanding Converged and Hyperconverged Environments

04:30pm New Generation of Storage Tiering: Less Management Lower Cost and Increased Performance

Wednesday 09:00am What Is Big Data? Architectures and Practical Use Cases

01:45pm Data Footprint Reduction – Understanding IBM Storage Efficiency Options

03:15pm IBM Spectrum Virtualize – SVC, Storwize and FlashSystem V9000 (repeats Friday)

Thursday 10:15am IBM Spectrum Scale and Elastic Storage Offerings

01:45pm IBM Spectrum Scale for File and Object storage

03:15pm IBM Storage Integration with OpenStack

05:45pm Storage -- Meet the Experts

Friday 10:15am IBM Spectrum Virtualize – SVC, Storwize and FlashSystem V9000



The Problem: Islands of Block, File and Object level data

Volume-level Storage

� OS-specific file systems on block-based devices

� Sharing requires file transfers

� Provides “Context” for Analytics of Social and Mobile transactions

File-level Storage

� NAS encourages sharing across social networks

� Desire for file sync-and-share across desktops and mobile

� HDFS requires transfer (ingest) from other sourcesJFS2

EXT4

NTFS

SMB

HDFS

NFS

Object-level Storage

� New Web and Mobile apps prefer Object-level access

Amazon S3

OpenStack

Swift



Volume vs. File vs. Object level access

POSIX

Read recordWrite record

Volume

Read blockWrite block

SAN orLAN

LAN orWAN

File


LAN orWAN

Object

Get, Put, Delete

NAS


HTTP

Get, PutDelete



Why use IBM Spectrum Scale™

Extreme Scalability

� Add or Remove nodes and storage, without disruption or performance impact to applications

Universal Access to Data

� All servers and clients have access to data through a variety of file and object protocols

High Performance

� Parallel access with no hot spots

Proven Reliability

� Used by over 200 of the top 500 Supercomputers

� Survive any node or storage failure with Distributed RAID and redundant components



IBM Spectrum Scale™ –Software, Systems or Cloud Services

Software

• Install software on your own choice of Industry standard x86 or POWER servers

Pre-built Systems

• Elastic Storage Server with Erasure Coding

• Storwize V7000 Unified

Cloud Services

• Spectrum Scale can be deployed on any Cloud

Scale



Spectrum Scale – Flexible File and Object Storage

FS1 FS256. . .Exabyte-Scale, Global Namespace

for files and objects

One big file system or divide into as many as 256 smaller file/object

systems

Each file system can be further divided into fileset containers

Flash and Disk LUNs are called Network

Shared Disks (NSD) Metadata can be separated to its own Pool or intermixed with

data

Files and objects can be migrated to Tape



IBM Spectrum Scale™ – Supported Topologies

Twin-tailed

SAN

Internal, Direct-Attach

Shared PoolsFPO Pools

NSD Servers

� Access files on direct, twin-tailed or SAN attached disk

� Can export files to application nodes

File Placement Optimization (FPO) Servers

� Access files on direct attached disk

� Exports files to other FPO servers

External Clients

� Access files via file and object protocols over IP network

TCP/IP

NSD Clients

� For Linux, AIX, and Windows

� Access files via SAN, TCP/IP or RDMA

TCP/IP or RDMA network



IBM Spectrum Scale™ replaces other POSIX file systems

SAN

Direct-Attach

SAN-level Storage

JFS2

EXT4

NTFS

• Works like OS-specific file systems• No file transfers required between OS• Linux on x86, POWER and z Systems

TCP/IP or RDMA Network

Twin-tailed



IBM Spectrum Scale™ – More than just a file system!

ROBO

Other NFS

Other Datacenters

Scale

Active File

Management

(AFM) caches data to where it is needed, can be used to migrate from other NFS

Hierarchical Storage

Management (HSM) migrates infrequently

accessed files to tape, automatically recalls back

when accessed

Local Read-Only Cache

(LROC) and Highly Available

Write Cache (HAWC) caches the busiest blocks of files on

local flash

Disaster Recovery

(DR) asynchronously mirrors data to remote

locations

Migrate/Recall Tape

NSD Client

Information Lifecycle

Management (ILM) moves data across tiers of flash

and disk



IBM Spectrum Scale™ – Local Read-only Cache

NSD Client

Local Flash on NSD Clients

� Automatically handles the flash cache so data is transparently available to your application with very low latency and no code changes

� Accelerates I/O performance up to 6x by reducing the amount of time CPUs wait for data

� Improves application performance while keeping all the manageability benefits of shared storage

Data is never stale

• Cache consistency ensured by standard tokens• Data is protected by checksum and verified on read• Write cache on two separate client nodes, or shared

fast storage device

Reduces Network Load

• Decreases the overall load on the IP network, benefitting performance for others

Local Read-Only Cache

(LROC) and Highly-Available

Write Cache (HAWC) caches the busiest blocks of files on

local flash



IBM Spectrum Scale™ – Policy Management

Migrate/Recall Tape

File Placement

� When new files are created, the active policy assigns it to the right pool – Flash, 15K, 10K or Nearline disk

� Files can be marked for having 2 or 3 replicas

� Files can be encrypted with specific keys

File Expiration

� Delete files automatically after they are no longer needed

File Movement (ILM)

� Move files between pools

� Based on age, size, heat, access frequently or other criteria

File Movement (HSM)

� Migrate files to an external pool of tapes

� Accessed files are automatically recalled back to internal pool of flash or disk

Hierarchical Storage

Management (HSM)

Information Lifecycle

Management (ILM)



IBM Spectrum Scale™ – Active File Management (AFM)

ROBO

Other Datacenters

Scale

Cloud

Global Namespace

� Shows entire file system, on flash, disk and tape, across all locations

Pre-fetch or Pull on demand

� Files can be periodically pre-fetched in advance, or pulled on demand when needed

WAN Caching

� Files you use most often are cached to your location for faster access and availability to avoid WAN delays

NFS Data Migration

� Use AFM to cache or migrate data from other NFS filers

Active File

Management

(AFM)Other NFS



IBM Spectrum Scale on any Cloud

Active File Management

Private VLAN

Scale

Scale

NSD Clients and Servers can be deployed within a Private VLAN on any Cloud



IBM Spectrum Scale™ – Backup and Disaster Recovery

Disaster Recovery

(DR) asynchronously mirrors data to remote

locations

Backup/Recover

FS1

snap1 snap2

Backup to External Media

• Files can be backed up to IBM Spectrum Protect, or third-party backup software

Asynchronous Mirror

• Use Active File Manager across data center locations

• Specify appropriate RPO

Snapshots

• Up to 256 Snapshots of entire file system, and 256 Snapshots of each file set

• Read-Only, Space-Efficient• Microsoft VSS Interface

• Writeable File Clones



Native Encryption and Secure Erase

Application

Remote Key Manager (RKM)

TCP/IP orRDMA

Node-to-Node Encryption

� Complies with NIST SP 800-131A

Data-at-Rest Encryption

• Files are encrypted by application node• Each file assigned random File-key• Master-key granularity by file or fileset, determined by

policies• RKM stores Master-keys, and nodes must have

appropriate RKM credentials• Data is encrypted from application node all the way to

NSD (flash or disk) media• FIPS 140-2 certified

Secure Erase

• Files are cryptographically erased by deleting their Master-key

• Files that “stay” are re-Mastered to new key



SAN

Twin-tailed

Protocol Servers

NFS v3/v4

SMB2 / SMB3

AIX, Linux, Mac OS, Windows, VMware,

z/OS, etc.

� Feature of IBM Spectrum Scale on

Linux nodes

� Share files with clients using NFS,

SMB and Object protocols

� All nodes can share the same data

� If Protocol Server Node fails client

connections are moved to another

server

� Protocol Server Node(s) need “NSD

Server” License

� External Clients need no Spectrum

Scale License

Clustered Protocol Servers for File and Object access

TCP/IP

OpenStack

Swift / S3



� No single-server performance and bottleneck scaling limits

- NAS filers can be a bottleneck and significantly impacts system performance

� No centralized metadata server

- Centralized metadata server can be a performance bottleneck for metadata intensive operations

TCP/IP Network

Network

File Server

Client

Nodes

Storage

Metadata

Data Data

Network

data

metadatadata Centralized

Metadata

Server

) (

) (

IBM Spectrum Scale™ is different thanother clustered/distributed storage solutions



NFS v3 versus NFS v4

Feature NFS v3 NFS v4

Exports Each export mounted separately

Pseudo File System combines all exports

State Stateless, no tracking of what clients open file

Stateful, server tracks which clients open/close

Firewall Multiple ports required for locking, status, etc.

Single port makes NFS v4 “firewall-friendly”

Security 32-bit integers for user and group identifiers

Strings@Domainenables Kerberos

Character set ASCII-7 requires all locales match

UTF-8 universal access

NFS v2 and v3 were originally developed by Sun Microsystems, which later turned over control to IETF for NFS v4



CIFS, SMB 2 and SMB 3

Feature SMB1, SMB2, SMB2.1 SMB 3

Availability Stateful retry Transparent FailoverSMB Scale-out

Connectivity TCP/IP, single channel TCP/IP or RDMA, SMB Multichannel

Performance Client reads directory metadata as needed

Client caches directory metadata with Leasing

Encryption None Data In-flight AES-128 between client / server

Snapshot Interface None Volume Shadow Copy Services (VSS)

CIFS – Part of Microsoft Windows NT 4.0 in 1996. SMB1 supersedes this version.SMB1 – Windows 2000, Windows XP, Windows Server 2003 and 2003 R2SMB2 – Windows Vista (SP1 or later) and Windows Server 2008SMB2.1 – Windows 7 and Windows Server 2008 R2SMB3 – Windows 8 and Windows Server 2012



IBM Spectrum Scale™ solves “Dropbox problem” for File Sync-and-Share

SAN


No IT Control:

• Servers and storage• Security• Access control • User provisioning• Sensitive data

TCP/IP or RFMA Network

Twin-tailed



HDFSNamenode

SecondaryNamenode

IBM Spectrum Scale™ – File Placement Optimization

SAN


• Spectrum Scale avoids the need for a central namenode, a common failure point in HDFS

• Avoid long recovery times in the event of namenode failure

• Spectrum Scale can consist of a mix of FPO and standard NSD servers, NSD client nodes, and Elastic Storage Servers (ESS) in the same cluster

File Placement Optimization (FPO)

Creates a “share nothing” cluster similar to HDFS in Hadoop environments

TCP/IP or RDMA



Hadoop Analytics – HDFS vs IBM Spectrum Scale™

HDFS Save

Results

Discardre

st

IBM Hadoop

Connector allows Map/Reduce

programs to process data without

application changes

IBM Spectrum ScaleApplication data stored on IBM

Spectrum Scale

is readily available for analytics

Save

Results

JFS2

NTFS

EXT4

Data Sources mashup of structured and unstructured data from a variety of sources

Actionable InsightsProvides answers to the

Who, What, Where, When, Why and How

Business Intelligence & Predictive Analytics> Competitive Advantages> New Threats and Fraud

> Changing Needs and Forecasting

> And More!



Share-Nothing versus Shared-Disk Deployments

DataData

Data Parity

DataData

Data

CopyCopy

Copy

CopyCopy

CopyTCP/IPor RDMA

Need more compute? Add another node!

Spectrum Scale and Elastic Storage Server reduce storage to one

RAID-protected copy of the data

Scale compute and storage capacity separately

Spectrum Scale FPO can keep 1,2 or 3

replicas of the data

Need more storage capacity?

Add another node!

3x versus 1.3x

TCP/IPor RDMA



SAN

Twin-tailed

Protocol Servers

NFS v3/v4

SMB2 / SMB3

AIX, Linux, Mac OS, Windows, VMware,

z/OS, etc.

� Feature of IBM Spectrum Scale on

Linux nodes

� Share files with clients using NFS,

SMB and Object protocols

� All nodes can share the same data

� If Protocol Server Node fails client

connections are moved to another

server

� Protocol Server Node(s) need “NSD

Server” License

� External Clients need no Spectrum

Scale License

Clustered Protocol Servers for File and Object access

TCP/IP

OpenStack

Swift / S3



One Dashboard – One Cloud

Compute• Hypervisors• Virtual Machines• VM images

Storage• Volumes• Objects• File Systems

Shared Services• Identity management

for users/passwords• Usage Statistics

Metering

Network• Focused on

TCP/IP based networks

Users, Developers, Administrators

OpenStack software controls large pools of compute, storage and networking resources

throughout a datacenter, managed through a dashboard or via the OpenStack API

Dashboard• GUI and CLI interfaces• Orchestration • Private Cloud

• Public Cloud



OpenStack terminology

Compute Storage Shared Services Network

Nova• Hypervisors• VM instances

Glance• VM images

• Disk images

Cinder• Volumes

Swift• Objects

Manila• File systems

Keystone• Policy and authentication

services, users and passwords

Ceilometer• Usage Statistics

Metering

Neutron• VPN• Firewall• Load

Balancing

Heat• Orchestration,

coordinate the deployment of resources for

an application

Horizon• OpenStack

dashboard, a web application that runs on Apache



Spectrum Scale supports OpenStack environments

Glance• VM images• Disk images

Cinder• Volumes

Swift• Objects

Manila• File systems

Global Name Space

Volume-on-file Object-on-file

• Create, Delete and Extend volumes• Take snapshots (FlashCopy) and clones• Volumes � Images, Images � Volumes• Attach and Detach to/from VM instances

• Create and Delete containers in account• Upload, Download and Delete objects• List containers or objects in a container• Display and update metadata

Keystone• Access control



OpenStack terminology – Storage

Cinder• Volumes

Swift• Objects

Manila• File systems Provides coordinated access to shared or distributed file

systems. The primary consumption of file shares would be across OpenStack Compute instances.

BlockLUN

Volume-on-file

Volume-on-object

Create, Delete and Extend volumes; take snapshots, images and clones; attach/detach from VM instances

Create and Delete containers and objects – Storing an object is like “valet parking” your data

Object-on-file

Object-on-database



Account / Container / Object architecture

Comparable to a mount point or root directory

Comparable to individual files

Comparable to a top level directory



Using OpenStack Swift



TCP/IP or RDMA

IBM Spectrum Scale™ for file and object clients

SAN

Protocol Nodes

Twin-tailed

Elastic

Storage

Server

TCP/IP Network

Network load balancer

Combined Proxy and Storage nodes

Storage nodes

Proxy nodes

OpenStack zones configured as Spectrum Scale Failure Groups

Object:OpenStack Swift

Swift S3



� Enables hybrid cloud data protection, without gateway appliances

� New storage pool options within IBM Spectrum Protect hierarchy

– On IBM SoftLayer cloud*

– On-premises native object stores, such as IBM Spectrum Scale

� Uses OpenStack Swift interfaces

Data Center

On-premises Storage Pools

Cloud Storage Pools

IBM Spectrum Protect Servers

Spectrum Protect -- New cloud and native object storage pools

* Other cloud services to be supported in future



The Solution: IBM Spectrum Scale™ brings it all together

Global Name Space

IBM Spectrum Scale™ replaces SAN-based file

systems

� Replaces NTFS, EXT4, JFS2 and other POSIX file systems

� Used by over 200 of the top 500 supercomputers

� No file transfers required between different OS

� Can be used with everything from databases to video streaming

� For x86, POWER andz System servers

� Secure with Data-at-rest encryption

IBM Spectrum Scale™ replaces HDFS and NAS file storage

� Full Hadoop interfaces for Map/Reduce analytics processing

� No transfer or ingest required as the data is already there

� Fully protected with Backup Software

� File-level access support for NFS, SMB, FTP, SCP and HTTPS

� Supports File Sync-and-Sharevia OwnCloud or Funambol

IBM Spectrum Scale™ offers Object access

� Object-level access based on OpenStack Swift and Swift S3 interfaces

IBM Spectrum Scale™ supports all media

� Spans flash, disk and tape media



IBM Software Defined Storage

Data Plane

IBM Software Defined Storage

Control Plane

IBM for Software Defined Storage

Control Protect

IBM ranked #1 in Software Defined Storage with

40% market share in 2014

Accelerate Virtualize Scale Archive

Universal access to data• Global Namespace with over 10

billion files

Proven Reliability• Introduced as GPFS in 1998• Over 1,000 production systems

High performance• Over 400 GB/sec throughput on

single system

Extreme scalability• Clusters with over 10,000 nodes• File systems with over 30 PB of data



Session Evaluations

YOUR OPINION MATTERS!

Submit four or more session evaluations by 5:30pm Wednesday

to be eligible for drawings!

*Winners will be notified Thursday morning. Prizes must be picked up at registration desk, during operating hours, by the conclusion of the event.

1 2 3 4





Many Features are Common to Swift and Spectrum Scale

Multi-tenant data access and

management

ACLs

Role-based Authentication

SSL/encryption

Multi-Region Geo-replication

High-Availability

Flat namespace

Storage automation

Simplified management

REST/HTTP

Mac/Windows/Linux

Swift/S3 API support

SDKs

User-defined metadata and search capabilities Extensible Swift

Middleware

Versioning

Quotas

Expiration

Rate Limiting

Rolling upgrades



IBM Tucson Executive Briefing Center

� Tucson, Arizona is home for storage hardware and software design and development

� IBM Tucson Executive

Briefing Center offers:

–Technology briefings

–Product demonstrations

–Solution workshops

� Take a video tour!

– http://youtu.be/CXrpoCZAazg



About the Speaker

Tony Pearson is a Master Inventor and Senior managing consultant for the IBM System Storage™ product line. Tony joined

IBM Corporation in 1986 in Tucson, Arizona, USA, and has lived there ever since. In his current role, Tony presents briefings

on storage topics covering the entire System Storage product line, Tivoli storage software products, and topics related to Cloud

Computing. He interacts with clients, speaks at conferences and events, and leads client workshops to help clients with

strategic planning for IBM’s integrated set of storage management software, hardware, and virtualization products.

Tony writes the “Inside System Storage” blog, which is read by hundreds of clients, IBM sales reps and IBM Business Partners

every week. This blog was rated one of the top 10 blogs for the IT storage industry by “Networking World” magazine, and #1

most read IBM blog on IBM’s developerWorks. The blog has been published in series of books, Inside System Storage:

Volume I through V.

Over the past years, Tony has worked in development, marketing and customer care positions for various storage hardware

and software products. Tony has a Bachelor of Science degree in Software Engineering, and a Master of Science degree in

Electrical Engineering, both from the University of Arizona. Tony holds 19 IBM patents for inventions on storage hardware and

software products.

9000 S. Rita Road

Bldg 9032 Floor 1

Tucson, AZ 85744

+1 520-799-4309 (Office)

[email protected]

Tony Pearson

Master Inventor,

Senior IT Specialist

IBM System Storage™



Email:[email protected]

Twitter:twitter.com/az99Øtony

Blog: ibm.co/Pearson

Books:www.lulu.com/spotlight/99Ø_tony

IBM Expert Network on Slideshare:www.slideshare.net/az99Øtony

Facebook:www.facebook.com/tony.pearson.16121

Linkedin:www.linkedin.com/profile/view?id=103718598

Additional Resources from Tony Pearson



Continue growing your IBM skills

ibm.com/trainingprovides a comprehensive portfolio of skills and careeraccelerators that are designed to meet all yourtraining needs.

If you can’t find the training that is right for

you with our Global Training Providers, we can help.

Contact IBM Training at [email protected]

Global Skills Initiative



Trademarks and Disclaimers

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.

Other product and service names might be trademarks of IBM or other companies. Information is provided "AS IS" without warranty of any kind.

The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Prices are suggested U.S. list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Photographs shown may be engineering prototypes. Changes may be incorporated in production models.

© IBM Corporation 2015. All rights reserved.References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at http://www.ibm.com/legal/copytrade.shtml.

ZSP03490-USEN-00