Intelligent Archiving Strategies: Toward ILM

Intelligent Archiving Strategies: Toward ILMArun Taneja, Founder and Consulting Analyst, Taneja GroupAlex Gorbansky, Senior Analyst, Taneja Group

Agenda A Bit of Historical Perspective Why Archive? What to Archive? The ILM Panacea Developing an Operational Archival Strategy Key Considerations Representative Vendors and Solutions Conclusions

Archival ≠ BackupBACKUP

Copying production data to an alternative medium for restorability in the event of data loss, corruption, or unavailability.

ARCHIVAL

Retention of historical data for future access for business reasons such as audits, customer issues, or litigation.

Some History On Archiving

3000 BCE

Ancient Egypt:• Library of Alexandria• Engravings

Middle Ages 1600s 1789 1884

Shift from Feudalism To Nation State:• Records• Property rights

American colonists• Births • Marriages• Businesses

French Revolution• Property records

American Historical Association• Archival standards • Marriages• Businesses

Archival Business Drivers TodayREGULATORY COMPLIANCE

REQUIREMENTS

EXPLOSIVE DATA GROWTH

APPLICATION PERFORMANCE DEGRADATION

RISING COSTS

What to Archive? Structured Data:

• ERP/CRM DB tiers• Business transactions

Unstructured Data: • Documents• X-Rays• Check Images• Voice recording

Semi-structured Data:• Email • Instant Messaging

ILM…ShmILM

“ILM” is an abstract framework for describing the processes and technology used to manage information throughout its life according to its business value.

“ILM” is NOT the panacea for your storage management challenges.

Archival is a key component of what vendors are calling “ILM

Applications: ERP, CRM, Email, Call Recording, Image Access

Application Data: Structured, Unstructured, Semi-Structured

Policies and RulesBusiness ContextReferential Integrity Regulatory Compliance

Snapshots HSMData Movement Technologies

Replication Backup Archival

PrimaryStorage Infrastructure Tiers

Secondary Tertiary

Developing an Archival StrategyPLAN When/How Data Classification Requirements

2. DESIGN

3. IMPLEMENT

4. REPORT &

TEST

Why Plan and When to Start Upfront Planning will Result in Significant Benefits in

Future Phases. Develop an Archival Strategy as part of your

application design and development process. Engage Key Stakeholders:

• Application Owners• Business Decision Makers: Compliance Officers, Legal

Identify Key Archival Business Drivers:• Regulatory Compliance• Other: Data Growth, Increasing Costs, Poor Performance

The Data Classification PuzzleAssess the application data in your shop

according to the following categories:• Structured: database• Unstructured: files, videos, images• Semi-structured: email

Identify specific data sets impacted by regulatory compliance:• Examples: Email, Medical Records, Call Recordings

Requirements DefinitionEngage Application OwnersCompliance not the ONLY archival driver Separate requirements processes for

applications impacted by compliance.Compliance-specific:

• Retention period• Media characteristics• Data restorability rates• Access control policies• Data availability/DR

General archival:• Data Access Patterns• Restore time requirements• Application performance• Cost structure• Access control policies• Data availability/DR

Taming the Compliance Monster1. Understand the Regulations: Significant Variance

by Industry

2. Assess/Communicate Requirements to Key Business Stakeholders

3. Judge Products for Yourself – Just because a vendor says a solution is “Compliant” doesn’t make it so.

4. Stay abreast of changes in regulatory mandates.

Defining Key Archival Metrics

Archive Distribution Percentages Across:• Online: Disk, Object-based storage• Near-line: Optical, Tape (local)• Off-line: Off-site vaults

Number of data copies• Local• Remote

Designing an Archival SolutionRequires an application specific

assessment – look for commonality in application requirements

Wholly enterprise-wide strategies will be difficult to build and sustain

Evaluate alternative solutions based on application requirements and metrics

Don’t Ignore the Organizational Dynamics

Archival Touches Multiple Organizations:• IT – Applications• IT – Infrastructure• Legal• Users

Consequences of mistakes are enormous:• Fines• Litigation

Consider organizing a cross-functional team led by an archival champion with a combination of technical and business expertise

Comprehensive Application Assessment

Data Classification Exercise Data Set Size and Historical and Predicted Data Growth Rates based

on business drivers Is Regulatory Compliance an Issue? Data Valuation over Time:

• Access patterns of data of 90 days old and beyond.• Cost of data loss

Going it alone can be difficult Available resources:

• Services organizations: GlassHouse, Accenture, EDS, Storage Vendor• Application Management Tools: File-Level SRM, Precise

Budgetary Requirements

Components of the Archival Stack

Application Specific ModuleDiscovery and analysis of data assets

Business rules and policies definitions

Identification and movement of specific data to

appropriate storage medium

Management, indexing of data and metadata

Access control mechanism

Application Data

Storage InfrastructurePhysical archive repository

Data Preservation and Protection

Indexing Technologies for Retrieval

Management

& Control

Physical

Repository

Data Flow

Structured Data Archival Challenges to Investigate

ERP deployments are still very nascent Preventing application downtime during archival Preserving referential data integrity:

• Archival of core data and associated data in other tables

Enforcing single read-only state across related data Delivering transparent access to archived/combined data via

native app UI• Maintaining performance of remote queries and union views.

Update process:• Restate vs. entire reload

Unstructured Data ConsiderationsScalability Sustained performance with data growth• Hierarchical file-systems limited at large scales

Content Access and VisibilityMeta data use to intelligently manage and

maintain archive addresses traditional file system limitations

Scalability of Index (Content addresses)

Email Archival Challenges

Stringent regulations: SEC Rule 17A-4• Non-rewriteable, non-reusable media• Verification of writes• Serialize units of media

Solution Requirements• Server-based capture• Support for multiple distributed Email Servers

Meta Data Holds Real Value

• Digital asset tied to specific infrastructure• No value outside of infrastructure context

• Self-describing attributes for digital asset• Enables powerful policy-based data

movement applications

Traditional File Systems

Object-based systems

Meta Data is data about data

Object Age and creation date Object Change History Associated application/users Access control Priority/Criticality Data Access/Frequency

Choosing the Right Storage Medium

Amount of D

ata

Probability of Reuse

D2D Systems

Libraries

Drives

< Seconds Minutes Hours to Days

1 WeekLife Expectancy

1 Month 3 Months 18 – 30 Years1 Year

Disk Systems

Recovery Time

Object Storage

Key Considerations for Storage Media

Cost Access time Application access method:

• NFS/CIFS• Application-specific API

Reliability/Availability Data Preservation Capability Scalability Archival solution integration

Storage Media ConsiderationsPros Cons

Primary Storage No risk of data lossInstantaneous access

Exorbitant costsPerformance degradation

Secondary Storage (SATA)

Cost effective Solid access time

IntegrationEnforcing preservationManagement

Object Storage Fit for large unstructured files Elimination of data redundancy WORM-like preservation

Price premiumPerformance scalability with index growth

Tape Most cost effective Removable Integrated WORM

Access timeReliability

Shifting towards an On-line Model

Primary

ObjectStorage

Tape

SATA

Representative VendorsStructured Data Email Unstructured

Archival Solutions

OuterBay, Princeton Softech, Applimation, Ixos,

Legato, KVS, Assentor Documentum, FileNet, NICE

SATA Object TapeStorage Platforms

CLARiiON, STK, IBM, Nexsan COPAN, Centera, Archivas, Permabit, DCT

STK, Quantum, ADIC, IBM

Start with your application vendor

Trust But Verify

Develop processes to periodically access historical data to test:• Data integrity• Access time

Manage capacity growth using vendor-supplied reporting tools

SummaryArchival is not backup and is not just about

complianceSuccessful strategy requires application-centric

approachEngage with key corporate stakeholders to

define requirements and select solutionsLook for automated and interoperable software

and hardware modules.Be Paranoid!

Thank you!

Arun Taneja

[email protected] Gorbansky

[email protected]