Mesh Services Architecture And Concepts Abolade Gbadegesin Principal Software Architect Microsoft Corporation BB0 6
Apr 01, 2015
Mesh Services Architecture And Concepts
Abolade GbadegesinPrincipal Software ArchitectMicrosoft Corporation
BB06
Live ServicesUser-centric data and services with more than 460M Users
Mesh
S
erv
ices
Identity Directory StorageCommunicatio
ns and Presence
Search & Geospatial
Users DevicesSynchronizatio
nApplications
Agenda
User Experiences Concepts Architecture Planning and Deployment Implementation Operations, Experiences, Lessons Learned Future Work
Using the Live Mesh software
demo
Key Capabilities
Provisioned state User, devices, applications, relationships
User-data storage Feeds and content Feed synchronization and content replication
Communications and presence Device presence and remote UI access Peer connectivity
Notifications Single channel for asynchronous push Signaling channel for session establishment
User-data storage
Accounts
Notifications
Communications& Presence
deviceclaim
device
identity
Provisioning In The MeshConceptual elements
,
provisioned during software install
provisioned during web sign-up
1
2
3
builds on Live ID authentication tocapture user-device relationships
Understanding REST In The MeshResources, collections and links
resource
collection
resource
UriEntityTagSubscriptionUri…
UriLastUpdateTime…
resource
linkUriLabel…
scope for grouping resourcesand tracking changes
metadata common to all resources
primitive for representing &navigating relationships
represented as feeds and items
1
2
3
4
Provisioned State ServicesConceptual view and building blocks
Mesh Service
Accounts Live Mesh Client
Accounts
Live ID Service Live IDClient Runtime Library
appapp
users, devices, apps (local cache)
HTTP
HTTP
mesh object
Folder Synchronization In The MeshConceptual elements
localfilesystem
data feed mapping
,
meshdatafeed
‘maps’ data feed to a file system location
1
2
items in data feedrepresent files
mesh object defines permission scope for sharing feeds
3
Understanding FeedSync In The MeshEnabling bidirectional feed synchronization
<rss version="2.0"> <channel> <title>To Do List</title> <description>A list of items to do</description> <link> http://example.com/partial.xml </link> <item>
<title>Buy groceries</title> <description>Get milk, ... </description> <sx:sync id="item_1_myapp_2005-05-21T11:43:33Z" updates="3">
<sx:history sequence="1" when="2005-05-21T09:43:33Z" by="a9"/> <sx:conflicts ...>
<item>...</item>
</sx:conflicts></sx:sync>
</item> </channel>
</rss>
version vector and conflicts for each feed item
localfile
system
Synchronized User-Data Storage ServicesConceptual view and building blocks
Mesh Service
User-data storage Live fx Client
User-data storage
Mesh folders app
,
(local replica)
010010010100111100
Scalable storage
feeds, content,permissions
HTTP, FeedSync
handles mesh folders
Mesh Services Architecture
Mesh Service
User-data storage
Live fx Client
User-data storage
010101
Scalable storage
Services fabric
Local file system
Accountsappapp
Microsoft Update
Live ID Service
Accounts Live IDClient
Runtime Library
MessengerNAT
Traversal
MessengerRelay
Service
Mesh UX
Communications& Presence
Notifications
Notifications
Communications& Presence
Mesh Folders
MeshWeb Desktop
MeshRemote Desktop
Derive workloads from dominant application flows
Identify key scalability factors Decide on the partitioning of state Assign machine roles across tiers
Mesh Services ArchitectureGetting from concepts to implementation
Mesh AccountsLive fx Client
Mesh Services PlanningWorkload analysis: Startup and file updates
update local cachesubscribe for change notifications
Startup
File change
Mesh User-Data Storage
synchronize feedupload updated files
synchronize local replicassubscribe for change notifications
Mesh NotificationsLive fx Client Mesh User-Data Storage
Mesh Services PlanningWorkload analysis: Content replication
fire change notifications
Contentreplication synchronize local replica
query availability of updated files
fetch updated files from peers
File change
continued
Mesh Communications and Presence
deliver change notifications
query availability of peers
relay or direct connection
Partition by user, device, and mesh object Use soft state to minimize I/O load Leverage HTTP 1.1 semantics for caching, change
notification, and incremental state transfer Leverage client-side resources for holding state Leverage peer connectivity for content replication
Mesh Services PlanningScale-out strategy
Mesh Services PlanningPartitioning resources
https://accounts.mesh.com/Identities/FKEXBSLJR3VUXEXMTPSSOVO6GM/DeviceClaims/CSOW2TEBYUSE5BD3I5QOS724UU
each resource lives in a ‘home’ datacenter
each resource has a stable canonical URI based on its resource identifier
each resource has a location URI that routes requests directly to its ‘home’ datacenter
https://accounts.clus9.bay.mesh.com/Identities/FKEXBSLJR3VUXEXMTPSSOVO6GM/DeviceClaims/CSOW2TEBYUSE5BD3I5QOS724UU
requests to the ‘wrong’ datacenter get redirected, and clients cache the redirect URI for future use
mapping a resource identifier to its ‘home’ datacenter requires a lookup in a replicated database
Mesh Services PlanningGeneral service design principles
Commodity hardware Partitioning for scaling out,
redundancy for availability Loose coupling across roles Xcopy deployment and configuration Fail-fast, recovery-oriented error handling Self-monitoring and self-healing
Mesh Services DeploymentInfrastructure and roles
Account APIs
Services fabric
User-Data Storage
Subscription store
Notificationqueue store
Device presence store
Dictionary state store
Partitioning & Recovery Manager
Watchdogs
front endroles
back endroles
infrastructureroles
commodity hardwarexcopy deployment
automated rollbackself-monitoring
fail-fast recoveryself-healing
scaleunit
Scalable storageServices fabric SKU:2P Dual Core8GB RAM4x 500GB SATA II HDDWindows Server
parti
tioni
ng u
nits
Web Desktop
Content upload & download
Front end roleBack end roleBuilding block
Services fabric
Mesh Services DeploymentScale unit layout
Subscription store
Notificationqueue store
Device presence store
Dictionary state store
Partitioning & Recovery Manager
Wat
chdo
gs
Web Desktop
Content upload & download
Notifications API
Accounts Scalable Storage
User-Data Structured
Storage
Live ID Messenger Relay Hotmail
User-Data Blob
Storage
reliable state
stores
soft state stores
building block services
Accounts API
User-Data Storage API
Communications & Presence
soft state stores
Front end roleBack end roleBuilding block
Communications and Presence
Mesh Services ImplementationSoft state partitioning and recovery
Node 1 Node 2
{ECCA8793-3208-48A9-A1A1-038D0AD704C5}
/MeshObjects/FJ3GZXUNDC4EPPPPUMQ2QNCG6I/Subscriptions/GLW6BC6HRWFU5LEW7HTX5FPC3Y
hash nodes & resourcesto the same output space
node ‘nearest’ a resourceis responsible for it
Notificationqueue store
Device presence store
Partitioning & Recovery Manager
Owner library
Lookup library
Owner library
Notifications API
Lookup library
consistent hashingnode loss tracking
resource owner nodes
resource client nodes
owner library obtains leaseslookup library queries assignments
Mesh Services ImplementationCommon software stack
Services fabric
controller
Mesh services host process
Windows Server
Mesh service applicationMesh services runtime
AP configuration
NETFX 3.5 SP1
Tracing & log
collection
multiple processes mesh service process
Mesh service codeBuilding block code
singleton process
SMB.sys
Mesh Services ImplementationFront end software stack
Services fabric
controller
Mesh services host process
Windows Server
HTTP.sys sockets
HTTP request dispatcher
TLS/TCP stream
dispatcher
Mesh resource handlers
Mesh content handlers
TLS/TCPhandlers
Mesh services runtime
AP configuration
Subscriptions
Soft state lookup
NETFX 3.5 SP1
Tracing & log
collection
SMB.sys
WCF syndication
Accounts
User-data storage
Communications & Presence
Notifications
Scalable storage client
User-data content
upload & download
TCP notification streaming
multiple processes mesh service process singleton process
Mesh service codeBuilding block code
Mesh Services ImplementationBack end software stack
Services fabric
controller
Mesh services host process
Windows Server
sockets
Sockets dispatcher
Mesh resource managersMesh services runtime
AP configuration
Subscriptions
Soft state owner/lookup
NETFX 3.5 SP1
Tracing & log
collection
SMB.sys
Scalable storage client
multiple process mesh service process singleton process
Work queues
Device presence manager
Notification queue manager
Subscription manager
Dictionary store manager
Partitioning and recovery manager
Mesh service codeBuilding block code
Monitoring The Mesh Services
demo
Mesh Services OperationsExperiences and lessons learned
On availability:Design for loosely coupled dependence on building blocksDiligently validate client/cloud upgrade scenariosInvest in pre-production stress and functional coverage in environments that look like productionDesign for throttling based on both dynamic thresholds and static bounds
Mesh Services OperationsExperiences and lessons learned
On monitoring:Continuously refine performance counters, logs, and log processing toolsMonitor end-user-visible operations (Keynote)Build end-to-end tracing across tiersSelf-healing is hard: Invest in tuning watchdogs and thresholds
Mesh Services OperationsExperiences and lessons learned
On deployment:Deployments every other week, client upgrades every monthMajor functionality roughly each quarterTook advantage of gradual ramp to learn lessons early
Mesh Services FuturesExtending the mesh cloud
synchronize user-data storage with non-Microsoft services
(e.g. Smugmug)
federate user authentication with other identity providers
(e.g. Active Directory)
personal and enterprise user-data storage
offerings:on-premises software
and online service
You saw this working today:
Microsoft Services
Connector
You saw this in the Sync Framework session
Stay tuned!
Microsoft Services ConnectorFederating identity
Desktop
Windows Azure
ISV Apps
Live Mesh
EnterpriseApps
Azure Platform Services
Microsoft Federation Gateway
Live Identity Provider
Exchange Your AppsSharepoint
ActiveDirectory
Enterprise On-PremisesMicrosoft Services
Connector
Online Services
CRM Online
Live Services
LiveMesh
Soon: enterprise credentials in Live Mesh
Future: enterprise policy on Mesh Services
and policy
Mesh Services
Sync Framework and Live Framework Synchronizing clouds
Mesh Services
User-data storage010101
Accountsappapp
Communications& Presence
Notifications
CloudSync
Live Framework
SyncFX store providers
Sync apps
Future: Mesh Services as a sync hub for your apps
The Mesh Services deliver a comprehensive and compelling suite of key capabilities
You can build on the Mesh Services today using the Live Framework and Sync Framework
Your feedback will help define the roadmap for delivering personal and enterprise offerings of the Mesh Services
Summary
Live Mesh http://www.mesh.com http://blogs.msdn.com/livemesh
Live Framework and Mesh Services BB04: Live Services: A Lap Around the Live
Framework and Mesh Services BB34: Live Services: Notifications, Awareness, and
Communications Sync Framework
http://msdn.microsoft.com/sync TL30: Microsoft Sync Framework Advances
Related Sessions and Resources
Evals & Recordings
Please fill
out your
evaluation for
this session at:
This session will be available as a recording at:
www.microsoftpdc.com
Please use the microphones provided
Q&A
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.