Azure Data Platform Jesus Aguilar Sr Technical Evangelist Microsoft @giventocode Microsoft Azure
Azure Data PlatformJesus AguilarSr Technical Evangelist Microsoft
@giventocodeMicrosoft Azure
Data Platform Components
A NoSQL document store that
offers fast time to market,
differentiated querying and
tuning capabilities with
schema-less web scale
Relational Non-Relational
SQL DB (PAAS) Table Storage File/ Blob StorageDocument DBSQL VM (IAAS)
SQL Database
Starting With The BasicsSQL Database
The Basics
SQL Server database technology as a service
Fully Managed
Enterprise-ready with automatic support for HA
Designed to scale out elastically with demand
Ideal for simple and complex applications
A Server Is Not A Machine
How It Works
Client Layer - Used by application to communicate directly with SQL Database.
Services Layer – Gateway between Client layer and Platform layer.
Platform Layer – Includes physical servicers and services that support the Services layer.
Infrastructure Layer – IT administration of the physical HW and OS.
PHPWCF Data
Services
SQL Server
Applications
and Tools
ODBC ADO.NET
Tabular Data Stream (TDS)
Server Provisioning
Service head that contains databases
Connect via automatically generated FQDN (xxx.database.windows.net)
Initially contains only a master database
Log on to Microsoft Azure Management Portal
Create a SQL Database server
Specify admin login credentials
Add firewall rules and enable service access
Use Microsoft Azure Platform PowerShell cmdlets (or use REST API directly)
wappowershell.codeplex.com
Selecting the right EditionService
Tier
Performance
Level
Common App
Pattern
Performance Business Continuity
Max DB
Size
Trans. Perf.
Objective
DTUs PITR DR / GEO-Rep
Basic Basic Small DB, SQL opp 2 GB Reliability / Hr. 5 Past 7
Days
DB Copy +
Manual Export
Standard S1 / S2 Wrkgp/cloud app,
multiple concurrent
operations
250 GB Reliability / Min. 15/ 50 Past 14
Days
DB Copy +
Manual Export
Premium P1 / P2 / P3 Mission Critical, High
volume, Many
concurrent Users
500 GB Reliability / sec. 100/
200/
800
Past 35
Days
Active Geo-
replication
DemoCreating A SQL Database Server
Create And Deploy
Your Database
Create Database…
Transact-SQL
Languages
.NET Framework (C#, Visual Basic, F#) via ADO.NET
C / C++ via ODBC
Java via Microsoft JDBC provider
PHP via Microsoft PHP provider
Frameworks
OData, Entity Framework, WCF Data Services, NHibernate
Tools
SQL Server Management Studio (2008 R2 and later)
SQL Server command-line utilities (SQLCMD, BCP)
CA Erwin® Data Modeler
Embarcadero Technologies DBArtisan®
Focus on logical vs. physical administration
Database and log files automatically placed
Three high-availability replicas maintained for every database
Tables require a clustered index
Maximum database size is 500 GB
Use command, distributed transactions, distributed views
Service Broker
Common Language Runtime (CLR)
SQL Agent
SQL Profiler
Native Encryption
Enhanced Tooling
Web designers for tables, views, stored procs
Interactive query editing and execution
Visual Studio IDE for database development
Includes modern designers and projects with declarative, model-driven development
Develop and test in both connected and disconnected states
Platform targeting for both SQL Server (2005 and above) and SQL Database
Get it free with Web PI, with SQL Server 2012 and with Visual Studio 11
Database Deployment
Alternative to traditional script based approach
Dramatically simplifies deployment, migration and versioning of databases
Provides a single unit of deployment for schema (dacpac) or for schema + data (bacpac)
Supports automatic versioning of database schemas
Supports platform targeting for both SQL Server (2005 and above) and SQL Database
Build from scratch or extract from existing db
With SQL Server Data Tools
With SQL Server 2012/2014 Management Studio
With SQL Database Import/Export Service
Via sqldacexamples.codeplex.com
Secure Your DatabaseSQL Database
There Are Two Ways To Secure A Database:
Server Benefits
SQL authentication supported (No Integrated authentication)
The Admin login is similar to sa
Connect to master to administer logins
loginmanager: Server-Level security role for creating logins
dbmanager: Server-Level security role for creating databases
Database Benefits
Logins require an associated user account
The Admin login is automatically associated with dbo
The dbo has full rights in the database
Manage users with CREATE / ALTER / DROP USER commands
Add users to roles via sp_add_rolemember to grant privileges
Utilize schemas where appropriate
SQL Database Firewall
• IP Address-based access control for SQL Database
• Rules can be defined at the server and database
• No IP authorized by default
• Configurable using the SQL Database Portal and REST API
• Option to disable/enable access from applications hosted in Microsoft Azure
Application Connectivity
1. TDS (Tabular Data Stream) protocol over TCP/IP supported
2. SSL required
3. Use firewall rules to connect from outside Microsoft data center
ASP.NET EXAMPLE:
1. login: [login]@[server]
2. Idle connections
3. Long running transactions
4. DoS guard
5. Failover events
6. Throttling
7. Connection pooling and Retry logic
8. Latency introduced for updates
9. No cross-database dependencies
<connectionStrings><addname="AdventureWorks"connectionString=
"Data Source=[server].database.windows.net;Integrated Security=False;Initial Catalog=ProductsDb;User Id=[login];Password=[password];Trusted_Connection=False;Encrypt=true;"
providerName="System.Data.SqlClient"/></connectionStrings>
Elastic SQL Database – Scaling out!
• .NET Client Libraries• Management of Shards
• Data Access
SQL on IaaS
Run SQL on VM
• Run any SQL product on cloud VM
• Support for SQL Server, Oracle, MySql
• Ready to go VM images available in Gallery
• Persistent storage using attached disk in blob storage
23Microsoft Azure
SQL Database vs SQL IaaS Comparison
24Microsoft Azure
Azure Storage Architecture
“Microsoft Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM
Symposium on Operating System Principals (SOSP), Oct. 2011
Azure Files
“I wish I could go to storage and provision a cloud drive, giving it a namespace, and that drive would then be UNC-addressable by the OSes.”
Azure Files – Customer Quotes
• Setup an IaaS VM to host a File Share backed by an IaaS Disk• Write code to find the IaaS File Share from the rest of the VMs in
your service.• Write some code to provide high availability
• Handle host upgrades, node failures
• You can only access the File Share from other VMs
Sharing Files – The old way
Azure Files•
•
•
Azure Files
Azure Files
Website Served From Azure File Share
Load Balancer
Azure
VM
Azure
VM
…
Blob Storage
Blob Storage Concepts
Blob Details
Blob Details
Blob Details
Blob Containers
• Special $root container
• A container holds a set of blobs• Set access policies at the container level • Associate Metadata with Container• List the blobs in a container• Including Blob Metadata and MD5 • NO search/query. i.e. no WHERE MetadataValue = ?
• Effectively in Partition of 1• Target of 60MB/s per Blob
Enumerating Blobs
• Prefix
• Delimiter
• Include= (snapshots, metadata etc…)
Pagination
• Either set maxresultsor;
• Exceed default value for maxresults (5000)
Two Types of Blobs Under the Hood
Uploading a Block Blob
Uploading a large blob
BenefitEfficient continuation and retry
Parallel and out of order upload of blocks
Microsoft Azure
Storage
Page Blob – Random Read/Write
Shared Access Signatures
• Use short time periods and re-issue
• Use container level policy that can be deleted
• Ad-hoc
• Policy based
Ad Hoc Signatures
• Signedresource Blob or Container
• AccessPolicy Start, Expiry and Permissions
• Signature HMAC-SHA256 of above fields
• Single use URLs
• E.g. Provide URL to mobile client to upload to container
Policy Based Signatures
• Specify StartTime, ExpiryTime, Permissions
• Signedresource Blob or Container
• Signedidentifier Optional pointer to container policy
• Signature HMAC-SHA256 of above fields
• Providing revocable permissions to certain users/groups
• To revoke: Delete or update container policy
NoSQL48
Generally scales more easily
• The storage engines of NoSQL stores are designed to minimize contentions enabling higher throughput and therefore more scalable
• Lower transaction capability in NoSQL results in less contention and therefore more scalable
• Less complex query processor means that a single query can’t degrade service
• Built-in replication capability means that store can scale out which better aligns to other application tiers (e.g. websites)
• No fixed schema or lower schema requirements
49Microsoft Azure
NoSQL on Azure
• Azure Tables service is NoSQL row store
• DocumentDB born in the cloud document database (JSON) and JS (PAAS).
• HBase is a Big Data (Hadoop) NoSQL store available in HDInsight
• MongoDB is a document (JSON) store
• Cassandra is a columnar store with excellent replication
50Microsoft Azure
Tables51
Table Storage Concepts
Table Details
Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order
Timestamp Optimistic ConcurrencyExposed as an HTTP Etag
No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double
No Fixed Schema
Querying
Purpose of the PartitionKey
Entity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction
Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountMicrosoft Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table
Partitions and Partition Ranges
DocumentDB
Key ConsiderationsNimble document database
Scalable document database as a service
Extensibility through JavaScript
Cross-Platform access
Transactions
Configurable Consistency Levels
Schema-Free
Document Indexing
Resource Model
Database
TenantCollection
Document
Attachment
User
/users/{id}
Permission
/dbs/{id}
/colls/{id}
/docs/{id}
/attachments/{id}
/permissions/{id}
Document
Service
/Addresses
/addresses
Items and Feeds (Nouns)Natively stored and served as JSON
URI addressable
Partitioned for scale-out
Replicated for HA
Granular access control
A subscription can provision 1+ Document Services
Can run in shared or dedicated mode
Each can have 1+ databases
Interaction ModelRESTful interaction over HTTP
Standard HTTP verbs & semantics
Interact using your favorite HTTP client
Built-in Support for TCP
Novel, efficient and powerful document centric query model
Javascript based sprocs/triggers /evals
POSTItem
resource TenantFeed URI
PUTItem
resource Item URI
DELETE Item URI
GET TenantFeed Or
Item URI
Create a new resource
/Execute a script
Replace an existing resource
Delete an existing resource
Read/Query an existing
resource
Update an existing resource
PATCH Item URIItem
resource
location headquarters exports
Belgium 0 1
city
Moscow
city
Athens
0
country city
Germany Berlin
1
country city
France Paris
0
headquarters exports
country city
Italy 0 1
Germany Bonn
city dealers
Berlin 0
city
Amsterdam
name
Hans
location
Azure
footprint
16 regions worldwide in 2014
Fortune 500 using Azure
>57% >250kActive websites
Greater than
1,000,000SQL Databases in Azure
>20TRILLIONstorageobjects >300 MILLION
AD users
>13 BILLIONauthentication/wk>2
MILLIONrequests/sec >1
MILLIONDevelopers
registered with
Visual Studio
Online
Get startedVisit azure.microsoft.com
SQL Database Billing Rates (As of February 2012)
Database Size Price Per Database Per Month
0 to 100 MB Flat $4.995
> 100 to 1 GB Flat $9.99
> 1GB to 10 GB $9.99 for first GB, $3.99 per additional GB
> 10 GB to 50 GB $45.954 for first 10 GB, $1.998 for each additional GB
> 50 GB to 150 GB $145.874 for first 50 GB, $0.999 for each additional GB
SQL Database Architecture