Slide 1
Storing and Accessing Live Mashup Content in the CloudKrzysztof Ostrowski, Ken Birman
Cornell University{krzys|ken}@cs.cornell.edu1Agenda2A new, versatile storage abstraction:Checkpointed Channel (CC)
A new web application architecture:A web of (hyperlinked) CCsAgenda3Introduction4
Storing Content in the Cloud
server in adata centerclient-serverinteractionscloudcontentedge5
Storing Content in the Cloud
server in adata centerclient-serverinteractionscloudcontentedge6
server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge7
server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge8
server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge9
server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge10
server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge11
Storing Content in the Cloudcontentclient-serverinteractionsserver in adata centerupdate relayed through the server12
server in adata centerStoring Content in the Cloudexplicit request,e.g. HTTP GETcontentcloudedge13http://liveobjects.cs.cornell.edu
14
server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicapeer-to-peerinteractions15
server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replica16
server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replica17
server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replica18
server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replicaupdated19
server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replicaupdatedmulticast20
server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicaupdatedupdatedupdated21
server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicareplicated statemachine22
server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicamanagemembership23
server in adata centerStoring Content at the Edgeedgedatacontrol24
server in adata centerStoring Content at the Edgeedgedatacontrol25Cloud vs. Edge?
server in adata centeredgepeer-to-peerinteractionscontent replicacontent replicacontent replica
server in adata centerclient-serverinteractionscloudcontentedge26
server in adata centerCloud vs. Edge?edge27
server in adata centerCloud vs. Edge?edge28
server in adata centerCloud vs. Edge?edge29
server in adata centerCloud vs. Edge?edgeno replicas left30
Cloud vs. Edge?
server in adata centerclient-serverinteractionscloudcontentedgeflow ofupdatesbottleneck31
Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedge
flow ofupdatesbottleneck32
Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedge
flow ofupdatesbottleneck33Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedge
flow ofupdates
bottleneck34Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedgeflow ofupdates
bottleneck
35Cloud vs. Edge?Server capacity:SecondLife: ~40 clients/serverSecond Life and the New Generation of Virtual Worlds.S. Kumar et al. 2008. IEEE Computer, 41(9):46-53
36Cloud vs. Edge?Server capacity:SecondLife: ~40 clients/server
virtualisland37Cloud Edge?
server in adata centeredgepeer-to-peerinteractionscontent replicacontent replicacontent replica
server in adata centerclient-serverinteractionscloudcontentedgepersistencescalability+and38
Extending Cloud to the Edgeserveredgecloud39
Extending Cloud to the Edgecloudserveredge40
peer-to-peerinteractionsclient-serverinteractionsExtending Cloud to the Edgecloudservercontentcontent41
peer-to-peerinteractionsclient-serverinteractionsExtending Cloud to the Edgecloudservercontentdistributedprotocol(client-server,peer-to-peer, etc.)content42
Extending Cloud to the Edgecloudserveredge43There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft
=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud44There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft
=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud45There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft
=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud46There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft
=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud47There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft
=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud48Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams
New Storage Abstraction49Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams
New Storage Abstraction50Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams
New Storage Abstraction51Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams
New Storage Abstraction52Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams
New Storage Abstraction53Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams
New Storage Abstraction54http://liveobjects.cs.cornell.edu
55Checkpointed Channels (CC)56
Architecture57
Architecture58
Architecture59Architecture
60Architecture
61Architecture
networkmessages62the edge may cut through the channel
content physically resides outside the channelbut it could also be cached in it63the edge may cut through the channel
content physically resides outside the channelbut it could also be cached in it64
content physically resides outside the channelbut it could also be cached in itthe edge may cut through the channel65
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates66
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates67
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates68
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates69
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates70
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates71
Dynamics (local)Checkpointed Communication Channel (CC)UCUUUinitial checkpoint72
Dynamics (local)Checkpointed Communication Channel (CC)Cinitial checkpointmultiple updatesUUUU73
Dynamics (local)Checkpointed Communication Channel (CC)Cinitial checkpointmultiple updatesUUUU74
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates75
Dynamics (local)Checkpointed Communication Channel (CC)UupdatesUU76
Dynamics (local)Checkpointed Communication Channel (CC)UupdatesUU77
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates78
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates
79
InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates80
Dynamics (local)Checkpointed Communication Channel (CC)requestcheckpointR81
Dynamics (local)Checkpointed Communication Channel (CC)requestcheckpointR82
Dynamics (local)Checkpointed Communication Channel (CC)Ccheckpoint83
Dynamics (local)Checkpointed Communication Channel (CC)Ccheckpoint84
Dynamics (global)Checkpointed Channel (CC)85
Dynamics (global)Checkpointed Channel (CC)connect86
Dynamics (global)Checkpointed Channel (CC)connectinitialize87
Dynamics (global)connectinitializeneedcheckpoint88
Dynamics (global)initializeneedcheckpointR89
Dynamics (global)request checkpointneedcheckpoint90
Dynamics (global)request checkpointR91
Dynamics (global)initializerequest checkpointneedcheckpointR92
Dynamics (global)request checkpointneedcheckpointprovide checkpoint93
Dynamics (global)request checkpointprovide checkpointCcheckpoint94
Dynamics (global)provide checkpointCcheckpoint95
Dynamics (global)Ctransfercheckpoint96
Dynamics (global)transfercheckpointC97
Dynamics (global)Ccheckpointtransfercheckpoint98
Dynamics (global)Ccheckpointdeliver checkpoint99
Dynamics (global)deliver checkpointCcheckpoint100
Dynamics (global)consumecheckpointCdeliver checkpoint101
Dynamics (global)consumecheckpointCwants toupdate102
Dynamics (global)consumecheckpointCwants toupdateupdateU103
Dynamics (global)wants toupdateupdateUrequestupdate104
Dynamics (global)updaterequestupdateU105
Dynamics (global)Upropagateupdate106
Dynamics (global)propagateupdateUUU107
Dynamics (global)propagateupdateUUUupdateU108
Dynamics (global)UUUupdateUdeliverupdate109
Dynamics (global)UUUUdeliverupdate110
Dynamics (global)deliverupdateUUUU111
Dynamics (global)UUUUconsumeupdate112SemanticsUCC'+=earlier checkpointor statenewcheckpointor stateincrementalupdate113SemanticsUCC'+=earlier checkpointor statenewcheckpointor stateincrementalupdate114SemanticsUCC'+=earlier checkpointor statenewcheckpointor stateincrementalupdate115
SemanticsCkcheckpointmultiple updatesU1U2Un116
SemanticsCkU1U2Un117
SemanticsCkU1U2UnCk+1+=CkU1118
SemanticsCkU1U2UnCk+1Ck+2+=Ck+1U2119
SemanticsCkU1U2UnCk+1Ck+2Ck+n+=Ck+n-1Un120
SemanticsCkU1U2UnCk+1Ck+2Ck+n121
SemanticsCkU1U2UnCk+1Ck+2Ck+n122
SemanticsC1U2U3U5C4C6123
SemanticsC1U2U3U5C4C6C2C3C5124
SemanticsC1U2U3U5C4C6C2C3C5125
SemanticsC1U2U3U5C4C6C2C3C5126
SemanticsC1CnC2C3mastersequence127
SemanticsC1CnC2C3mastersequenceC1Un-1U1U2submitted updates128
SemanticsC1CnC2C3mastersequenceC1C5C8C9C15every application observes a subset of the master sequence129
SemanticsC1CnC2C3mastersequenceC1C5C8C9C15every application observes a subset of the master sequenceand nobody lags behind130TypesCheckpointed Communication Channel (CC)can be classified based on:the type of checkpoints,the type of updates,(and many other factors we wont dicuss)131TypesCheckpointed Communication Channel (CC)can be classified based on:the type of checkpoints,the type of updates,(and many other factors we wont dicuss)132
TypesCUXML documentsstandardized edits on XML documentsXML Channel (XC)133
TypesCUXML documentsstandardized edits on XML documentsXML Channel (XC)134
TypesCUstandardized edits on XML documentsXML Channel (XC)XML documents135
TypesCUXML Channel (XC)XML documentsstandardized edits on XML documents136TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection137TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection138TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection139TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection140http://liveobjects.cs.cornell.edu
141Live Distributed Objects (LO)142Ordinary Web ApplicationsdisplayUI components
143Ordinary Web ApplicationsdisplayUI components
144Ordinary Web ApplicationsdisplayUI components
145
Ordinary Web ApplicationsdisplayUI components146
Ordinary Web ApplicationsdisplayUI components147Ordinary Web ApplicationsdisplayUI components
148Ordinary Web Applications
side by sidenested149Ordinary Web Applications
150Live Objects Applicationsside by sidenested
front-end to back-end151Live Objects Applicationsside by sidenestedfront-end to back-end
datapipeline152Live Objects Applicationsside by sidenestedfront-end to back-end
datapipelineUI component153Live Objects Applicationsside by sidenestedfront-end to back-end
datapipelineUI componentMPEG-2 decoder154Live Objects Applicationsside by sidenestedfront-end to back-end
datapipelineUI componentMPEG-2 decoderAES decryptor155Live Objects Applicationsside by sidenestedfront-end to back-end
datapipelineUI componentMPEG-2 decoderAES decryptorreliable transport156Live Objects Applicationsside by sidenestedfront-end to back-end
datapipelineUI componentMPEG-2 decoderAES decryptorreliable transportunreliable transport157
side by sidenestedfront-end to back-end
datapipeline158
side by sidenestedfront-end to back-end
159
side by sidenestedfront-end to back-end
160
side by sidenestedfront-end to back-end
161
side by sidenestedfront-end to back-end
162
side by sidenestedfront-end to back-end
163
Live Objects Applications
UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy164
Live Objects Applications
UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy165
Live Objects Applications
UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy166
Live Objects Applications
UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy167
Live Objects Applications
UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy168
Live Objects Applications
datapipeline169
Live Objects Applications
datapipeline
datapipeline170
Live Objects Applications
datapipeline
datapipelinedatapipeline171
Live Objects Applications
XML172References
XMLactivatechannelreferencechannelchannelproxy173References
XMLactivatechannelreferencechannelchannelproxy174References
XMLactivatechannelreferencechannelchannelproxy175References
XMLchannelreferencechannelchannelproxypoints to176References
XMLchannelreferencechannelchannelproxypoints tochannelcontentXMLXML177References
XMLpoints to
points toXMLXML
178References
XMLpoints to
points toXMLXML
179References
XMLXMLXMLXML
XML
XMLXMLXML
XML
XMLXML
XML
XMLXML
XML
XMLXML180Ordinary Web Applications
side by sidenestedXML markup181
Live Objects Applications
XMLXMLXML
XMLXMLXML
XMLXML
182
Live Objects Applications183
Live Objects Applications
space184
Live Objects Applications
space
plane 1XML185
Live Objects Applications
space
plane 1
plane 2XMLXML186
Live Objects Applications
space
plane 1
plane 2
building 1XMLXMLXML187
Live Objects Applications
space
plane 1
plane 2
building 1
mapXMLXMLXMLXML188
Live Objects Applications
tile 1XML189
Live Objects Applications
tile 1XML
tile 2XML190Live Objects Applications
191Live Objects Applications
desktop192Live Objects Applications
desktop
text 2XML193Live Objects Applications
desktop
text 2XML
image 1XML194
Live Objects Applications
desktop
text 2XML
image 1XMLdesktop 2XML195
Live Objects Applications
desktop
text 2XML
image 1XMLdesktop 2XML
text 3XML196
Live Objects Applications
desktop 1197
Live Objects Applications
desktop 1XML
desktop 2198
Live Objects Applications
desktop 1XML
desktop 2
desktop 3XML199
Live Objects Applications
desktop 1XML
desktop 2
desktop 3XML
text 1XMLXMLXML200
Live Objects Applications
desktop 1XML
desktop 2
desktop 3XML
text 1XMLXMLXML201Conclusions202Cloud and edge technologies can work togetherbut we need a new storage abstraction.Checkpointed Channels (CCs)Support a variety of protocols: client-server, P2P, distributed replication, append log files, etc.,Can be used at any level of the protocol stack, at the fronted and at the backend, Can efficiently store structured mashup content,Can be hyperlinked into webs of CCs; these could serve as a basis for a new Web architecture.Conclusions203Thanks204