Windows Azure Storage & SQL Azure Eric D. Boyd Founder and CEO, responsiveX Windows Azure MVP [email protected] @EricDBoyd – EricDBoyd.com
May 20, 2015
Windows Azure Storage & SQL Azure
Eric D. BoydFounder and CEO, responsiveX Windows Azure MVP
[email protected]@EricDBoyd – EricDBoyd.com
Agenda
Storage Overview
Getting started
Working with Blobs
Adding Tables and Queues
Build and Deploy your Database
Core Services
Caching CDN
Service Bus Reporting
Data Sync Azure Connect
Identity HPC
Additional Services
Windows Azure
Compute Storage Database
Windows Azure Storage
Storage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service usesInstant Concurrency
Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the Internet
Wrapped in API by SDKMicrosoft.WindowsAzure.StorageClient
Windows Azure Storage Abstractions
TablesStructured storage. A table is a set of entities; an entity is a set of properties.
QueuesReliable storage and delivery of messages for an application.
BlobsSimple named files along with metadata for the file.
DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.
Windows Azure Storage Account
Create account in Management Portal
http://windows.azure.com
Geographically located with affinity
You get 2 keys…
API based on REST…
http://btlod.blob.core.windows.net/public/helloworld.txt
http://<account>.<BLOB|TABLE|QUEUE>.core.windows.net/...
Tools to Explore Storage Many tools are available some for free, some from 3rd parties, including:
Visual Studiohttp://myAzureStorage.cloudapp.nethttp://clumsyleaf.comhttp://cerebrata.com
These give access to your storage account
Storage Accountsdemo
Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 26+ global CDN nodes
Can co-locate storage account with compute accountExplicitly or using affinity groups
Accounts have two independent 512 bit shared secret keysAllows for key rotation when one is compromised
Limit of 100 TB of data per storage accountEach subscription defaults to five storage accounts
Call tech support to create more
Storage in the Cloud Emulator
Storage Emulator provides a local “Mock” storage
Emulates storage in cloud
Allows offline development
Requires SQL Express 2005/2008 or above
SSMS Provides local view of storage items
http://msdn.microsoft.com/en-us/gg433135
Storage Account
Storage AccountSLA – 99.9% AvailabilityCapacity – Up to 100 TBs per storage accountTransactions – Up to 5,000 requests per secondBandwidth – Up to a few hundred megabytes per second
Single PartitionThroughput up to 60 MB/sUp to 500 transactions (entities or messages) per second
Go Beyond the LimitsPartition between multiple storage accounts and partitionsWhen limit is hit, app may see ‘503 server busy’: applications should implement retry.
Working with BlobsBinary Large Objects include resources we use in our applications like pictures, videos, html, css, etc. that may not fit best in a relational schema
Choice of API’s – REST vs. SDK wrapper To use SDK add references to Microsoft.WindowsAzure.StorageClient Create Storage Account Object Add client for blobs Work with container references and blobs
Table Storage Concepts
EntityTableAccount
contoso
Name =…Email = …
Name =…EMailAdd=
customers
Photo ID =…Date =…
photos
Photo ID =…Date =…
Table Details
InsertUpdate Merge – Partial update
Replace – Update entire entity
UpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction
Create, Query, DeleteTables can have metadata
Not an RDBMS! Table
Entities
Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order
Timestamp Optimistic ConcurrencyExposed as an HTTP Etag
No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double
No Fixed Schema
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
FAV SPORT
Canoeing
Querying
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
?$filter=Last eq ‘Wegner’
Purpose of the Partition KeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction
Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
BikesSuper Duper Cycle
… 2009
BikesQuick Cycle 200 Deluxe
… 2007
… … … …
Canoes Whitewater … 2009
Canoes Flatwater … 2006
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
Rafts14ft Super Tourer
… 1999
… … … …
SkisFabrikam Back Trackers
… 2009
… … … …
Tents Super Palace … 2008
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
BikesSuper Duper Cycle
… 2009
BikesQuick Cycle 200 Deluxe
… 2007
… … … …
Canoes Whitewater … 2009
Canoes Flatwater … 2006
Rafts14ft Super Tourer
… 1999
… … … …
SkisFabrikam Back Trackers
… 2009
… … … …
Tents Super Palace … 2008
Partitions and Partition Ranges
Server A
Table = Products
Server B
Table = Products[Canoes - MaxKey)
Server A
Table = Products[MinKey - Canoes)
Loosely Coupled Workflow with QueuesEnables workflow between rolesLoad work in a queueProducer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps) Use multiple queuesRead messages in batchesMultiple work items per message
Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
Queue Details
Simple asynchronous dispatch queueNo limit to queue length, subject to storage limit.64kb per messageListQueues - List queues in account
Queue operations CreateQueue DeleteQueueGet/Set MetadataClear Messages
Message operationsPutMessage– Reads message and hides for time periodGetMessages – Reads one or more messages and hides themPeekMessages – Reads one or more messages w/o hiding themDeleteMessage – Permanently deletes messages from queueUpdateMessage – Clients renew the lease and contents
Queue’s Reliable Delivery
Guarantee delivery/processing of messages (two-step consumption)
Consumer dequeues message and it is marked as invisible for a specified “Invisibility Time”
Consumer deletes message when finished processing
If consumer crashes, message becomes visible for another consumer to process
Message lifecycle
Queue
Msg 1
Msg 2
Msg 3
Msg 4
Worker Role
Worker Role
PutMessage
Web Role
GetMessage (Timeout)
RemoveMessage
Msg 2Msg 1
Worker Role
Msg 2
Windows Azure Storage SummaryFundamental data abstractions to build your applicationsBlobs: Files and large objectsDrives: NTFS APIs for migrating applicationsTables: Massively scalable structured storageQueues: Reliable delivery of messages
Easy to use via the Storage Client Library
Core Services
Caching CDN
Service Bus Reporting
Data Sync Azure Connect
Identity HPC
Additional Services
Windows Azure
Compute Storage Database
Start with the SQL Azure Basics
SQL Azure Database isSQL Server database technology delivered as a service on the Windows Azure PlatformIdeal for both simple and complex applicationsEnterprise-ready with automatic support for HADesigned to scale out elastically with demand
Get started quicklyChoose a planChoose a billing optionProvision servers
Ready to get started?Let’s jump into development
Provision Your ServerServer definedService head that contains databases
Connect via automatically generated FQDN (xxx.database.windows.net)
Initially contains only a master database
Provision servers interactivelyLog on to Windows Azure Management Portal
Create a SQL Azure server
Specify admin login credentials
Add firewall rules and enable service access
Automate server provisioningUse Windows Azure Platform PowerShell cmdlets (or use REST API directly)
wappowershell.codeplex.com
SQL Azurecreate server & db
demo
Build Your DatabaseUse familiar technologiesSupports Transact-SQL
Supports popular languages.NET Framework (C#, Visual Basic, F#) via ADO.NET
C / C++ via ODBC
Java via Microsoft JDBC provider
PHP via Microsoft PHP provider
Supports popular frameworksOData, Entity Framework, WCF Data Services, NHibernate
Supports popular toolsSQL Server Management Studio (2008 R2 and later)
SQL Server command-line utilities (SQLCMD, BCP)
CA Erwin® Data Modeler
Embarcadero Technologies DBArtisan®
Differences in comparison to SQL Server Focus on logical vs. physical administration
Database and log files automatically placed
Three high-availability replicas maintained for every database
Databases are fully contained
Tables require a clustered index
Maximum database size is 150 Gb
Unsupported SQL Server featuresBACKUP / RESTORE
USE command, linked servers, distributed transactions, distributed views, distributed queries, four-part names
Service Broker
Common Language Runtime (CLR)
SQL Agent
Build Your DatabaseThin client database developmentIntroducing the SQL Azure management portal
Web designers for tables, views, stored procs
Interactive query editing and execution
Rich client database developmentIntroducing SQL Server Data Tools (SSDT)
Visual Studio IDE for database development
Includes modern designers and projects with declarative, model-driven development
Develop and test in both connected and disconnected states
Platform targeting for both SQL Server (2005 and above) and SQL Azure
Get it free with Web PI, with SQL Server 2012 and with Visual Studio 11
Deploy Your DatabaseData-tier Application Framework (DAC Fx)Alternative to traditional script based approach
Dramatically simplifies deployment, migration and versioning of databases
Provides a single unit of deployment for schema (dacpac) or for schema + data (bacpac)
Supports automatic versioning of database schemas
Supports platform targeting for both SQL Server (2005 and above) and SQL Azure
Build from scratch or extract from existing db
How to get the latest DAC FxWith SQL Server Data Tools
With SQL Server 2012 Management Studio
With SQL Azure Import/Export Service
Via sqldacexamples.codeplex.com
Deploy Your DatabaseInteractive approach for dacpac v1 and v2Manage your server with SQL Azure Management Portal
Deploy dacpac
Load data (scripts, bulk copy, SSIS)
Also supported in SQL Server 2012 Management Studio
Interactive approach for bacpac v2Upload bacpac to blob storage
Log into Windows Azure Management Portal
Import bacpac, table data automatically loaded
Also supports export to blob storage
Upgrading a dacpac or bacpacNot supported in portals yet
Use SQL Server 2012 Management Studio
There are two ways to secure a database:
Within the Database
On the Server
Connect Your ApplicationConnecting to SQL Azure1. TDS (Tabular Data Stream) protocol over
TCP/IP supported
2. SSL required
3. Use firewall rules to connect from outside Microsoft data center
ASP.NET EXAMPLE:
Special considerations1. Legacy tools and providers may
require special format for login: [login]@[server]
2. Idle connections terminated after 30 minutes
3. Long running transactions terminated after 24 hours
4. DoS guard terminates suspect connections with no error message
5. Failover events terminate connections
6. Throttling may cause errors
7. Use connection pooling and implement retry logic to handle transient failures
8. Latency introduced for updates due to HA replicas
9. No cross-database dependencies, result sets from different databases must be combined in application tier
<connectionStrings><addname="AdventureWorks"connectionString=
"Data Source=[server].database.windows.net;Integrated Security=False;Initial Catalog=ProductsDb;User Id=[login];Password=[password];Encrypt=true;"
providerName="System.Data.SqlClient"/></connectionStrings>
Visualize Your DataSQL Azure Reporting isSQL Server Reporting Services technology delivered as a service on the Windows Azure Platform
Ideal for operational reporting against SQL Azure data
Enterprise-ready with automatic support for HA
Designed to scale elastically with demand
Get started quicklyProvision report server via Windows Azure Management Portal
Build reports with Reporting Services Report Designer
Deploy reports to report server
Render reports with Visual Studio ReportViewer controls
Synchronize Your Data
SQL Azure (Hub)
SQL Azure (US)
SQL Azure (WE)
SQL Server
(OnPrem)
SQL Azure Data Sync isMicrosoft Sync Framework technology delivered as a service on the Windows Azure Platform
Ideal for scheduling synchronization between data sets hosted in SQL Azure or SQL Server
Uses a hub and spoke topology
Special considerationsConflict resolution policy configured centrally (hub or client wins)
Sync direction configured between each client and the hub (to hub, from hub, bi-directional)
Sync schedule must be between 5 minutes and 1 month
Data sets include multiple tables and can be filtered, triggers are added to data set tables
Tables added to hub and client schemas
Agent must be installed for on-prem clients
Scale Out Your Data
SQL Azure Federations providesIntegrated database sharding that can scale to hundreds of nodes
Multi-tenancy via flexible repartitioning
Online split operations to minimize downtime
Automatic data discovery regardless of changes in how data is partitioned
Special considerationsA logical database can contain multiple federations
Distribution scheme supports int, bigint, guid, and varbinary types
Filtering routes connection to appropriate shard regardless of changes in partitions
Merge, fan-out queries and automatic distribution of schema changes not supported in initial release
Common Data
Db
Federation Root
A-D E-M T-WN-S X-Z
Hands on Labbuild the backend
one and one half hours
lab 2
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after
the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.