K.Mohamed Faizal Lead Consultant, NCS Pte Ltd Microsoft Cloud Day Building web applications with Azure Storage Faizal has over a decade of experience in Information Technology with a focus on enabling portals, Internet & Intranet application development. In this session you will learn storage capabilities of Windows Azure, Blobs, Tables and Queues. Discover how to create storage accounts; upload and retrieve blobs and blob metadata; create, update and query tables; and create a simple service that uses a message queue for communication.
66
Embed
MS Cloud Day - Building web applications with Azure storage
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
KMohamed Faizal Lead Consultant NCS Pte Ltd
Microsoft Cloud Day
Building web applications with Azure Storage
Faizal has over a decade of experience in Information Technology with a focus on enabling portals Internet amp
Intranet application development
In this session you will learn storage capabilities of Windows Azure Blobs Tables and Queues Discover how to create
storage accounts upload and retrieve blobs and blob metadata create update and query tables and create a
simple service that uses a message queue for communication
CL201 Building Web Application with Azure Storage
KMohamed FaizalLead Consultant NCS (P) Ltd
27th April 2011
About Me ndash K Mohamed Faizal
15
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
CL201 Building Web Application with Azure Storage
KMohamed FaizalLead Consultant NCS (P) Ltd
27th April 2011
About Me ndash K Mohamed Faizal
15
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
About Me ndash K Mohamed Faizal
15
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Table Inserts Single Partition (SP) vs Multiple Partitions (
Table Get Single Partition (SP) vs Multiple Partitions (MP)
Are Unique Partition Key Values Sufficient To Scale
Slide 67
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale