Intro to AWS: Storage Services

©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

Intro to AWS: Storage Services

Guy Farber, AWS Storage Business Development

AWS storage options

• Scalable object storage

• Inexpensive archive storage

• Persistent block storage

• Shared file system

• Turnkey gateway solution

We are constantly producing more data

From all types of industries

Object Storage

Amazon S3

Simple Storage Service

2012 2013 2014 2015

102% year-over-year increase in

data transfer to and from S3

(Q4 2014 vs Q4 2013, not including Amazon use)

S3 usage

99.999999999%durability

Nice try kid…

Reduced redundancy option99.99% saves ~20%

S3 resources: buckets and objects

S3 website: static content

1 PB raw storage

800 TB usable storage

600 TB allocated storage

400 TB application data

S3 capacity pricing—pay only for what you use!

Amazon S3

S3 continuous cost reduction

• Available through 11 regions

globally

• Priced at per GB-month rates

• 8 price reductions since launch

• 51% average S3 capacity fee

reduction on 4/1/2014

• TCO: comparing on-premises to

S3

– Can be challenging for some

customers

– We can help!

S3

Amazon EMR

Amazon S3—New Features

S3 event notifications

Delivers notifications to Amazon SNS, Amazon SQS, or AWS

Lambda when events occur in S3

S3

Events

SNS topic

SQS queue

Lambda function

Notifications

Foo() {…}

• Preserve, retrieve, and restore every version

of every object stored in your bucket

• S3 automatically adds new versions and

preserves deleted objects with delete markers

• Easily control the number of versions kept by

using lifecycle expiration policies

• Easy to turn on in the AWS Management

Console

S3 versioning

S3 cross-region replicationAutomated, fast, and reliable asynchronous replication of data across AWS regions

Source

(Virginia)

Destination

(Oregon)

• Only replicates new PUTs. Once

S3 is configured, all new uploads

into a source bucket will be

replicated

• Entire bucket or prefix based

• 1:1 replication between any 2

regions

• Versioning required

Use cases:

• Compliance—store data hundreds of miles apart

• Lower latency—distribute data to regional customers)

• Security—create remote replicas managed by separate AWS accounts

Cross Region Replication Demo

S3 use cases

• Web-scale storage capacity and performance for web

applications

• Single-origin store with delivery through Amazon

CloudFront

• Staging and persistent store for Big Data applications

• Storage target for backup and active archive

Amazon Glacier

Low-cost archiving service

$0.01per GB-month

$120per TB/year

99.999999999%durability

3–5 hoursdata retrieval

Storage costs

vs.

retrieval costs

Glacier benefits

• Reduce cost for long-term archiving

• Leverage unlimited storage capacity

• Replace tape museums

• Improve durability

Amazon S3 – Glacier integration

Policy-based archiving service

S3 lifecycle policies →

Key prefix “logs/”

Transition objects to Glacier 30 days after creation

Delete 365 days after creation date

<LifecycleConfiguration>

<Rule>

<ID>archive-in-30-days</ID>

<Prefix>logs/</Prefix>

<Status>Enabled</Status>

<Transition>

<Days>30</Days>

<StorageClass>GLACIER</StorageClass>

</Transition>

<Expiration>

<Days>365</Days>

</Expiration>

</Rule>

</LifecycleConfiguration

SoundCloud—leveraging Glacier for

audio transcoding

• World’s leading social sound

platform

• Audio files must be transcoded and

stored in multiple formats

S3

Glacier

Use cases for S3 lifecycle policies

• Data tiering in the cloud

• Manage object versioning for logical data protection

• Deletion of data from Glacier by policy

Block Storage

Instance storage

Amazon EBS

Elastic Block Storage

1GB16 TB

EBS

General Purpose (SSD)

Up to 16 TB

10,000 IOPS

Up to 160 Mbps

Provisioned IOPS (SSD)

Up to 16 TB

20,000 IOPS

Up to 320 Mbps

Price Performance

EBS

Magnetic General Purpose Provisioned IOPS

Use cases Infrequent data access

Boot volumes

Small to med DBs

Dev and Test

I/O intensive

Relational DBs

NoSQL DBs

Storage media Magnetic disk-backed SSD-backed SSD-backed

Max IOPS 40–200 IOPS 10,000 IOPS 20,000 IOPS

Latency (random

read)20–40 ms 1–2 ms 1–2 ms

Availability Designed for 99.999% Designed for 99.999% Designed for 99.999%

Price$.05/GB-month

$.05/million I/O$.10/GB-month

$.125/GB-month

$.065/provisioned IOPS

Amazon EBSsnapshots

EBS use cases

• Persistent block storage for Amazon EC2

• Transactional workloads

• File system for an instance—NTFS, ExtFS, and so on

Introducing Amazon Elastic File System

Cloud-based file system

What is EFS?

• Fully managed file system for EC2 instances

• Provides standard file system semantics

• Elastically grows to petabyte scale

• Delivers performance for a wide variety of workloads

• Highly available and durable

simple elastic scalable

1 2 3

EFS is designed for a broad range of use

cases, such as…

• Content repositories

• Development environments

• Home directories

• Big Data

EFS is simple

• Fully managed– No hardware, network, or file layer

– Create a scalable file system in seconds!

• Seamless integration with existing tools and apps– NFS v4—open and non-proprietary

– Standard file system semantics

– Works with standard OS file system APIs

• Simple pricing = simple forecasting– EFS price: $0.30/GB-month

1

EFS is elastic

• File systems grow and shrink

automatically as you add and remove

files

• No need to provision storage capacity

or performance

• You pay only for the storage space you

use, with no minimum fee

2

• File systems can grow to petabyte

scale

• Throughput and IOPS scale

automatically as file systems grow

• Consistent low latencies regardless

of file system size

• Support for thousands of concurrent

NFS connections

EFS is scalable3

How does it all fit together?

AVAILABILITY ZONE 1

REGION

AVAILABILITY ZONE 2

AVAILABILITY ZONE 3

VPC

EC2EC2

EC2

EC2

Customer’s file

system

Why does this matter?...

…to app owners

and developers?

…to your

business?

• Easy to move existing code/applications/tools used

today with existing NFS servers to the AWS cloud

• Simple shared file storage solution for new cloud-native

applications

• Predictable pricing with no up-front investment

• Increased agility

• Spend less time managing file storage and more

time focusing on your business

…to IT

administrators?

• Eliminates need to manage and maintain file system

storage at scale

Amazon Storage Gateway

Storage Gateway

Your on-ramp to AWS cloud storage:• Back up into S3

• DR on EC2

• Archive into Amazon Glacier

• iSCSI or VTL interface

Summary: the Amazon storage portfolio

S3• Object storage: data presented as buckets of objects

• Data access by using APIs over the Internet

EFS• File storage (analogous to NAS): data presented as a file system

• Shared low-latency access from multiple EC2 instances

Glacier• Archival storage: data presented as vaults/archives of objects

• Lowest-cost storage, infrequent access by using APIs over the Internet

Storage

Gateway

• Back up and archive data into S3 and Amazon Glacier

EBS• Block storage (analogous to SAN): data presented as disk volumes

• Lowest-latency access from single EC2 instances

SAN FRANCISCO