Page 1
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Choosing the Right Data Storage Solution
Joe Lyons – Sr. Manager Global Storage
November 13, 2013
Page 2
AWS Storage Options
• Scalable Storage
• Inexpensive Archive Storage
• Persistent Direct Attached Storage
• Turn-key gateway solution
• Customer Presentation: iMemories
Page 3
We are constantly producing more data
Page 4
From all types of industries
Page 5
#1 Object Storage
●○○○○
Page 6
AMAZON S3 SIMPLE STORAGE SERVICE
Page 7
99.999999999% Durability
Page 8
Trillions Of Unique Customer Objects
Q4 2006
Q1 2007
Q2 2007
Q3 2007
Q4 2007
Q1 2008
Q2 2008
Q3 2008
Q4 2008
Q1 2009
Q2 2009
Q3 2009
Q4 2009
Q1 2010
Q2 2010
Q3 2010
Q4 2010
Q1 2011
Q2 2011
Q3 2011
Q4 2011
Q1 2012
Q2 2012
Q3 2012
Q4 2012
Q1 2013
Q2 2013
Q3 2013
Page 9
1.5 Million+ Peak Transactions Per Second
Page 10
Storage Tiers: Buckets + Unlimited Objects
Page 11
Reduced Redundancy Option 99.99% saves ~20%
Page 12
Spotify adds over 20,000 tracks a day - RRS
Page 13
Amazon S3 Website: Static Content
Page 14
Amazon S3 continuous cost reduction
• 16 Price reductions since launch
• TCO: On-premises vs. Amazon S3
– Can be challenging for some customers
– We can help!
Page 15
1PB Raw Storage
800TB Usable Storage
600TB Allocated Storage
400TB Written Application Storage
Amazon S3 -
Only actual
usage is
charged
RAW Storage at On perm vs. Cloud
Storage
Page 16
Use Amazon S3 When You Need
• Unlimited storage capacity
• High durability
• Storage for backups
• Single Origin Store with Delivery via Amazon CloudFront
Page 17
AMAZON GLACIER LOW-COST ARCHIVING SERVICE
Page 18
1c Per GB / Month
Page 19
$120 Per TB / Year
Page 20
99.999999999% Durability
Page 21
3-5 Hours Data Retrieval
Page 22
STORAGE COSTS
VS
RETRIEVAL COSTS
Page 23
Use Amazon Glacier When You Need
• Inexpensive/Long-term archiving
• Unlimited storage capacity
• Eliminated Tape Museums
• Eliminate Tech Refresh
• High durability
Page 24
Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE
Page 26
Archive Recovery Process with Tape
+ Days or Weeks
Page 27
Archive Recovery Process with AWS &
Amazon Glacier
$$
Hours
Glacier S3 EC2
/HPC
CloudFront Generating
Business
Value
Page 29
Use Amazon S3 and Amazon Glacier When You Need
• HSM in the cloud
• Archive data from Amazon S3/RRS to Amazon Glacier by policy
• Delete data from Amazon Glacier by policy
Page 30
#2 Block Storage
●●○○○
Page 31
#3 SYNC VOLUMES
●●●○○
Page 32
AMAZON EBS ELASTIC BLOCK STORAGE
Page 33
Ephemeral Storage
Page 35
IOPS Provisioned
4000
IOPS
Page 39
Amazon EBS Snapshots
Page 42
Use Amazon EBS When You Need
• Long-term persistent storage
• Data changes frequently
• Block storage for your databases – Provisioned IOPS volumes
• Filesystem for an instance NTFS, ExtFS, RAID, LVM…
• Access to raw, unformatted block-level storage
Page 43
AWS STORAGE GATEWAY
Page 44
AWS Storage Gateway
Page 45
What is AWS Storage Gateway?
45 Amazon Confidential
Integrates on-prem IT environments with Cloud storage for departmental and remote office backup and DR
Utilizes a virtual appliance that sits in customer datacenter
Exposes compatible iSCSI interface on front end
Stores primary data on-AWS in Amazon S3 or on-premise with data backed-up to Amazon S3 as Amazon EBS snapshots
Page 46
Solution Overview
Run apps in the cloud using your uploaded data –
HPC/Hadoop/Analytics
2. DR
1. Offsite Backup
3. Data Mirroring
AWS Storage Gateway works with your existing backup
application and moves your data into Amazon S3 as EBS
snapshots
Run AWS Storage Gateway in Amazon EC2 and access
snapshots up to 32 TB in size
AWS Storage Gateway – Cached Volumes will help you
create Storage Volumes in cloud and keep most recent
accessed data locally [Reduce SAN footprint for File Shares] 4. Dept. File Share
VTL version of AWS Storage Gateway would help customer
move data in the form of Virtual Tapes from on premise to
Amazon S3 (VTL – Virtual Tape Library) and then to Amazon
Glacier (VTS – Virtual Tape Shelf)
5. Archive/Glacier
Page 47
IT’S ALL ABOUT
CHOICE PERFORMANCE-ORIENTED
COST-ORIENTED
Page 48
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Understanding AWS Storage Options
Tamara Monson, iMemories, Inc.
November 13, 2013
Page 50
Our Story: History • iMemories was founded in 2006
• Cloud was in its infancy – AWS officially launched in 2006: Amazon
S3
• Handling physical assets: – VHS, Betamax, MiniDV tapes
– 16mm, 8mm film reels
– Photos, slides
• Using on-premises equipment: – VCRs, decks
– Film projectors
– Photo and slide scanners
• Focus was digitization for web/online viewing, DVD creation, archiving
2006
2008
20
10
2012
2013
iMemories' Growth
Page 51
Our Story: Success • Growing infrastructure
• Reliability not as high as desired
• Less-than-ideal customer experience
• Tripled full-time resources to manage
• Costs for static, infrequently accessed assets much higher than for more-frequently accessed assets
• Rate of growth driving need for infrastructure elasticity
2006
2008
20
10
2012
2013
iMemories' Growth
8TB
0.75PB
1.5PB
Page 52
Our Story: Need
Use Case
• Video and photo
converted and digital
source files
• Short-term local copy
• Access infrequently
– Disc creation
– Additional encodings
Requirements
• Durable, redundant,
secure
• Keep forever
• No or low operational
costs
• No new hardware
• Off-premises
Page 53
Evaluating Options: Calculating TCO
• Operations
– Percentage FTE
– Facilities and power
– Maintenance
• Hardware
– Life span
– Number of copies
– Raw GB vs. usable GB vs.
consumed GB
0
0.2
0.4
0.6
0.8
1
1.2
Time
Raw Usable Consumed
Page 54
Evaluating Options: Comparisons
Amazon S3 Cloud Files
Cloud Amazon Glacier
Page 55
iMemories Evolution
Architecture 1.0
• All storage and processing on-premises (circa 2006)
• Centered on physical assets, devices, servers
• Expensive: scale by adding more
• Non-ideal customer experience
Architecture 2.x
• Solve scale challenges
• Reduce costs
• Best possible customer
experience
Page 56
Storage 1.0
• On-premises
• ~1.5 PB capacity
• Limitations of scale:
– Add new hardware
– Space and power
File Mgmt
iMemories Scottsdale Data center
Digitization,
Encoding
Servers Storage Volume
Page 57
Storage 2.0: Amazon Glacier
AWS Cloud
File Mgmt
iMemories Scottsdale Data Center
Digitization,
Encoding
Servers Storage Volume
Glacier
Manager
Regions
Vaults
Amazon SQS Amazon SNS
Page 58
Amazon Glacier
Manager
- Interfaces with Amazon SQS
- Tracks restore/submit request
state
- Processes Submit requests in
real-time
- Processes Restore requests
once/hour
- 5% of consumed / 720 hours
- peak hourly rate
- Distribute request across 1-hr
windows
- Uses 10 MB chunks
- Performs audits 3x/week
- One discrepancy
- Zero lost files on Glacier (!!!)
- Generates internal activity reports
Glacier Manager
File Management
Digitization,
Encoding
Servers
Page 59
2.x: Move to AWS
Regions
Elastic Load
Balancing Video
Files
Availability Zones
Streaming
Servers
Streaming
Server
v-ec2-east1.imemories.com
v-ec2-west2.imemories.com
File Mgmt
iMemories Scottsdale Data Center
Digitization,
Encoding
Servers
Storage
Volume
S3 Manager
AWS Cloud
Page 60
2.x: Move to AWS
Region
Elastic Load
Balancing
Video
Files Upload
Web App
File Mgmt
iMemories Scottsdale Data Center
API
AWS Cloud
Amazon
SWF
u-ec2.imemories.com Video
Processing
Group
Page 61
In Closing
• iMemories Story – History
– Success
– Need
• Evaluating Storage Options – Define use case and solution requirements
– Understand TCO
– Design and build for Amazon Glacier’s 4-hr download SLA, 5% restore cap
• iMemories Evolution – Amazon Glacier is first full production integration
– Migrating more functionality to cloud
– Will always have an on-premises component
Page 62
Enabling families to share memories in endless ways.
http://www.imemories.com
[email protected]
Page 63
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
STG101