WINDOWS AZURE STORAGE – ARCHITECTURE VIEW Chaowlert Chaisrichalermpol Senior Developer Jetabroad
Jan 14, 2015
WINDOWS AZURE STORAGE – ARCHITECTURE VIEW
Chaowlert Chaisrichalermpol
Senior Developer
Jetabroad
AGENDA
Blob
Table
Queue
Storage services
MYSELF
Chaowlert Chaisrichalermpol
Azure Storage experiences
Architect of Samsung Gift & AIS Privilege
Monthly Active Users > 400,000
Daily Active Users > 150,000
Experienced migrate from Sql Azure to Table Storage
BLOB
BLOB – OVERVIEW
Store files
Files Media files Logs, raw data (consider using Azure Hadoop to analyze) Backup Any!
Cost is insanely cheap! (low as 24$/mth for 1TB!) This price include fault tolerance & API 48$/mth for geo replication
BLOB – DEMO
Movie provider
BLOB – BEST PRACTICES
Performance Point to Blob directly to reduce web role load Gzip text file before put into blob Resume download by Accept-Ranges header
Scalability Use CDN Scalability target per blob is 64M/sec Upload blob simultaneously is faster
TABLE
TABLE – OVERVIEW
NoSql key-value store
NoSql = No join + no grouping + no order by + no paging + no index + no transaction + no view + no stored proc + no function + no constraint + no default + no trigger + no fancy data type + no backup restore + much more!!!
TABLE – COMPARE WITH SQL AZURE
Azure Table SQL Azure
Features Basic Full sql features
Performance 20,000 rows/secHigher for cross accounts
180 concurrent connectionsHigher for premium services
Scalability Automatic Sql Federation
Availability Geo redundant Sql Data Sync
Cost (150 GB)
20$ (include 120M storage transactions)
226$ (no premium services, no federation, no data sync)
Usage Insert intensive data All others
TABLE – DEMO
Table Load Test
loader.io20 web roles
Azure Table
SQL Azure
TABLE – PERFORMANCE
Azure Table Sql Azure
Insert (DateTime Key) 2,113 tx/sec 588 tx/sec
Insert (Guid key) MAX!! 595 tx/sec
Update to single item 58 tx/sec 74 tx/sec
Read single MAX!! MAX!!
Read 1,000 179 tx/sec 860 tx/sec
TABLE – BEST PRACTICES
Review PartitionKey design Batch: Insert, update, delete up to 100 entities Ordering: Data always sorted by PartitionKey and RowKey
Scalability Scalability target for single partition is 2,000 rows/sec Avoid prepend/append pattern on PartitionKey Avoid hotspot on PartitionKey
Performance Use Json over AtomPub (70% bandwidth reduction) Use SCL 3.0+ (higher performance + use Json as default)
QUEUE
QUEUE – OVERVIEW
Persistent job processing
Able to process up to 32 messages at a time Not for log computing
Message size is up to 64kb May have to keep data in blob
QUEUE – VS SERVICE BUS
Azure Queue Service Bus
Max Age 7 days Unlimited
Protocols REST REST, SBMP, AMQP
Queue access Polling Polling, Long polling, Push
Feature Simple queue Session, Transaction, Duplicate detect, Auto dead lettering
Cost 0.05$ / million tx 1$ / million messages
Usage Job Messaging
QUEUE – HOW IT WORKS
Queue
Web Role
1. Add message
Worker Role
Message
2. Get message & lease
3. Delete message
QUEUE – SCENARIOS
Workload leveling Image resizing Order processing
Workload distribution Email sending Sending push notification
Temporal decoupling Night report jobs
QUEUE – DEMO
Ticket sell 300,000 users in 1 minute! Only 3,000 tickets available
QUEUE – BEST PRACTICES
Scalability Scalability target per queue is 2,000 message /sec Consider use multiple queues for higher scalability
Workload scaling AutoScaling by using MessageCount on queue
Cost of queue is worker role 99.9% worker role, 0.1% storage If usage is low, use web role as worker Use AutoScaling
STORAGE SERVICES
STORAGE – AVAILABILITY
LRS ZRS GRS RA-GRS
Store 3 repl single faculty
3 repl on multiple faculties
6 repl on 2 regions (3 repl each)
6 repl on 2 regions (3 repl each)
Durability Disk/node failure
Zone failure (fire)
Regional disaster
Regional disaster
Services Blob, Table, Queue
Block Blob only
Blob, Table, Queue
Blob, Table, Queue
Failover to secondary region
- - Auto within 15 mins
Failover read access by app
Price (first TB)/GB/month
$0.024 $0.030 $0.048 $0.061
STORAGE – SCALABILITY
Understand single partition unit
Blob Single Blob = Account + Container + Blob name Single Blob scalability target is 60MB/sec
Table Single Table Partition = Account + Table name + Partition Key Scalability target is 2,000 entities (1kb) / sec
Queue Single queue = Account + Queue name Single queue scalability target is 2,000 small message
(1kb)/sec
STORAGE – SCALABILITY
Scalability Target per account Capacity: up to 200TB Transaction: 20,000 messages (1kb) / second Network
Local redundant: Ingress 10Gibps/ Egress 15Gibps Geo redundant: Ingress 5Gibps/ Egress 10Gibps
Scale beyond single storage account
User Location Account
Lee Hong Kong Storage East
John London Storage USA
Amm Thailand Storage SE
STORAGE – PERFORMANCE
Storage & Web Role should be in the same region
Client can access storage directly (Public or SAS)
Use latest SDK
Net setting ServicePointManager.UseNagleAlgorithm = false; ServicePointManager.Expect100Continue = false; ServicePointManager.DefaultConnectionLimit = 100; (Or
More)
STORAGE – SECURITY
Public access (blob only) Direct access CDN access
Shared Access Signature (SAS) For client applications (mobile, web) Restricted access by permission: list, read, add, update,
delete Restricted access at data level Invalidate when regenerate full access key
Full access key For trusted services
STORAGE – DEMO
Todo list app (Share Access Signature )
ClientWeb Role
1. Request token
Azure Table
2. Access Data
Q & A