Outdoor Distributed Outdoor Distributed Computing Computing Liviu Iftode Liviu Iftode Department of Computer Science Department of Computer Science Rutgers University Rutgers University http://discolab.rutgers.edu http://discolab.rutgers.edu Supported by NSF through ITR grant ANI-0121416 Supported by NSF through ITR grant ANI-0121416
64
Embed
Outdoor Distributed Computing Liviu Iftode Department of Computer Science Rutgers University Supported by NSF through ITR grant.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Computing distributed for performance or fault tolerance Computing distributed for performance or fault tolerance Nodes are computationally equivalentNodes are computationally equivalent Configuration is stable (failures are exceptions)Configuration is stable (failures are exceptions) Networking is robust and has acceptable delaysNetworking is robust and has acceptable delays Relatively easy to programRelatively easy to program
Message passing or shared memoryMessage passing or shared memory Naming easy to translate to network addressesNaming easy to translate to network addresses Correct programs complete and return deterministic resultsCorrect programs complete and return deterministic results
Distributed object Distributed object tracking over a largetracking over a largegeographical areageographical area
Cars collaborating Cars collaborating for a safer and for a safer and more fluent trafficmore fluent traffic
Outdoor Distributed SystemsOutdoor Distributed Systems
Nodes are volatile and/or mobile: dynamic configurations Nodes are volatile and/or mobile: dynamic configurations Networking is wireless and ad-hoc: unpredictable delaysNetworking is wireless and ad-hoc: unpredictable delays Location dictates the computation: naming must include locationLocation dictates the computation: naming must include location How to program and execute distributed applications outdoors?How to program and execute distributed applications outdoors?
Traditional distributed computing does not workTraditional distributed computing does not work
How to Program Outdoor Distributed How to Program Outdoor Distributed Embedded Systems?Embedded Systems?
Recent research in networked embedded Recent research in networked embedded
systemssystems HardwareHardware Operating SystemsOperating Systems Network ProtocolsNetwork Protocols Data collection/dissemination in sensor networksData collection/dissemination in sensor networks
Our research focuses on programmabilityOur research focuses on programmability How to code distributed applications to execute How to code distributed applications to execute
over networks of embedded systems?over networks of embedded systems?
Traditional Distributed Computing Traditional Distributed Computing Does Not Work Well over Ad-HocDoes Not Work Well over Ad-Hoc
Fixed address naming and routing (e.g., IP) Fixed address naming and routing (e.g., IP)
are too rigidare too rigid
End-to-end data transfer may hardly End-to-end data transfer may hardly
novel system architectures and novel system architectures and
programming modelsprogramming models
ExampleExample
““Water the hottest spot on the Left Hill”Water the hottest spot on the Left Hill” Number and location of mobile sprinklers is Number and location of mobile sprinklers is
unknownunknown Configuration is not stable in timeConfiguration is not stable in time
sprinklers movesprinklers move temperature changestemperature changes
What is a good result?What is a good result?
Mobile sprinkler with temperature sensor
Left Hill Right Hill
Hot spot
Quality of ResultQuality of Result
QoR
Outdoors Adversity0 100%
100% ideal
real
Talk at a GlanceTalk at a Glance
Outdoor Programming ModelOutdoor Programming Model
Outdoor Execution ModelOutdoor Execution Model
ApplicationsApplications
Outdoor ProgrammingOutdoor Programming
– How to program an unknown number of mobile systems to execute an intrusion detection application in a certain physical space?- Programming model must be simple to use - Networking aspects should be hidden from programmer- Access to mobile systems as simple as access to variablesAccess to mobile systems as simple as access to variables- A virtual address space for the physical space?
Left Hill Right Hill
Sequential Programming ExampleSequential Programming Example
struct temp a[N];for(i=0;i<N;i++) {
if (a[i].temp> Max_temp) {Max_temp = a[i].temp;Max_id = i; }
}a[Max_id].water = ON;
Distributed Programming using Distributed Programming using Message PassingMessage Passing
Struct temp a[N/P], b[P];for(i=0;i<N/P;i++) {
if (a[i].temp> Max_temp) {Max_temp = a[i].temp;Max_id = i; }
Applications access distributed data through shared Applications access distributed data through shared variablesvariables
Runtime system translates variable accesses into Runtime system translates variable accesses into message passing (when necessary)message passing (when necessary)
int Max_temp, Max_id,i;for(i=k*N/P; i< (k+1)*N/P; i++) {
if (a[i].temp> Max_temp) {Max_temp = a[i].temp;Max_id = i; }
}b[k]=Max_temp;barrier()if (k==0) {
<calculate b_max and b_max_id>
From Indoor to Outdoor From Indoor to Outdoor ComputingComputing
Virtual Address Space Space Region
Variables Spatial References
Variables mapped to physical memory
Spatial references mapped to systems embedded in the physical space
Reference consistency
Bounded access time
?
?
Spatial Programming (SP) at a Glance Spatial Programming (SP) at a Glance
Outdoor distributed applications are Outdoor distributed applications are
programmed using spatial referencesprogrammed using spatial references
A virtual address space hides networking A virtual address space hides networking
Embedded systems/nodes named by their Embedded systems/nodes named by their
expected locations and propertiesexpected locations and properties
Space RegionSpace Region
radius
Hill = new Space({lat, long}, radius);
{lat,long}
Virtual representation of a physical spaceVirtual representation of a physical space Similar to a virtual address space in a Similar to a virtual address space in a
conventional conventional
computer systemcomputer system
Spatial ReferencesSpatial References
– Defined as {space:property} pairs–Similar to variables in conventional programmingSimilar to variables in conventional programming
–Used to name systems embedded in physical space
–Indexes used to distinguish among similar systems in the same space region
-Same index maps to the same system (reference consistency)-Distinct indexes map to distinct systems
{Hill:robot[0]}
{Hill:robot[1]}
{Hill:truck}
Hill
Reference ConsistencyReference Consistency
At the time of the first access, a spatial reference At the time of the first access, a spatial reference
is mapped to an unmapped embedded system is mapped to an unmapped embedded system
located in the specified space located in the specified space
Mappings maintained in per-application Mapping Mappings maintained in per-application Mapping
Tables (MT)Tables (MT)
A spatial reference refers the same system as A spatial reference refers the same system as long as it is located in the same space, fails long as it is located in the same space, fails otherwiseotherwise
Smart Message (SM)Smart Message (SM) User-defined distributed applicationUser-defined distributed application Executes on nodes of interest named by Executes on nodes of interest named by
propertiesproperties Migrates between nodes of interestMigrates between nodes of interest
Self-RoutingSelf-Routing Application-controlled routing to perform Application-controlled routing to perform
migrationmigration Executed on every node on the path between Executed on every node on the path between
nodes of interestnodes of interest Cooperative NodesCooperative Nodes
Execution environment (Virtual Machine)Execution environment (Virtual Machine) Memory addressable by names (Tag SpaceMemory addressable by names (Tag Space
Smart vs. “Dumb” MessagesSmart vs. “Dumb” Messages
Prevents SMs from migrating to nodes where Prevents SMs from migrating to nodes where they cannot executethey cannot execute
SMs specify resources requirementsSMs specify resources requirements Lower bounds for virtual machine cycles, tag space Lower bounds for virtual machine cycles, tag space
memory, runtime memory, network bandwidth, etc.memory, runtime memory, network bandwidth, etc. Specified by programmer or derived using compiler Specified by programmer or derived using compiler
supportsupport SMs accepted if the node can satisfy SM SMs accepted if the node can satisfy SM
requirementsrequirements SMs transfer only the missing code bricksSMs transfer only the missing code bricks More resources can be granted according to More resources can be granted according to
admission policyadmission policy If not granted, SM is allowed to migrateIf not granted, SM is allowed to migrate
Smart Message ExecutionSmart Message Execution
Takes place over a virtual machineTakes place over a virtual machine
Non-preemptive but time boundedNon-preemptive but time bounded
Ends with migration or terminatesEnds with migration or terminates
During execution, SMs canDuring execution, SMs can Create new SMs Create new SMs Spawn new SMsSpawn new SMs Access the tag spaceAccess the tag space Block on a tags to be updatedBlock on a tags to be updated
Tag SpaceTag Space Local memory that SMs can useLocal memory that SMs can use
SM-created tags: limited lifetime, persistent across SM SM-created tags: limited lifetime, persistent across SM
executionsexecutions
I/O tags: maintained by the systemI/O tags: maintained by the system
I/O port access I/O port access read (temperature)read (temperature)
Store dataStore data write (tag,value)write (tag,value)
Inter -SM communication Inter -SM communication
Synchronization on writeSynchronization on write block(tag,timeout)block(tag,timeout)
Routing Routing
Protection Domains for Tag SpaceProtection Domains for Tag Space
Owner: SM that creates the tagOwner: SM that creates the tag Family: all SMs having a common ancestor with the SM Family: all SMs having a common ancestor with the SM
that creates the tagthat creates the tag Origin: all SMs created on the same node as the family Origin: all SMs created on the same node as the family
originator of the tag owner originator of the tag owner Code: all SMs that carry the same code brick(s)Code: all SMs that carry the same code brick(s) Others: all SMsOthers: all SMs
FamilyFamily
CodeCodeOriginOrigin
Others
OwnerOwner
Access Control Example Access Control Example (Code-based protection domain) (Code-based protection domain)
discovery, image capture, light sensor, system statusdiscovery, image capture, light sensor, system status
Prototype Node with GPS receiver and video camera
Lightweight Migration Lightweight Migration Traditional process migration difficultTraditional process migration difficult
Strong coupling between execution entity and hostStrong coupling between execution entity and host OS state (e.g., open sockets, file descriptors) hard to transferOS state (e.g., open sockets, file descriptors) hard to transfer
Tag space decouples the SM execution state from the Tag space decouples the SM execution state from the
OS stateOS state
SM migration transfersSM migration transfers Data bricksData bricks explicitly specified by programmer as mobile explicitly specified by programmer as mobile
datadata Minimal execution control stateMinimal execution control state required to resume the SM at required to resume the SM at
the next node (e.g., instruction pointer, operand stack the next node (e.g., instruction pointer, operand stack
pointer) pointer)
Experimental Results for Simple Experimental Results for Simple Routing AlgorithmsRouting Algorithms
Completion Time
Routing algorithmCode not cached (ms)Code cached (ms)
Geographic
On-demand
415.6 126.6
506.6 314.7
user nodenode of interestintermediate node
Self-Routing SimulationSelf-Routing Simulation
starting nodestarting node node of interest other nodeother node
3 nodes of interest located in the corners have to be visited in clockwise order vary the radius from 100m to 700m
0
0.5
1
1.5
2
2.5
3
0 100 200 300 400 500 600 700 800
Region Radius (meters)
Com
plet
ion
Tim
e (s
ec)
On-Demand Routing Geographic+On-Demand Routing
0
500
1000
1500
2000
2500
3000
3500
4000
4500
0 100 200 300 400 500 600 700 800
Region Radius (meters)
Byt
es S
ent
in t
he
Net
wo
rk (
KB
ytes
)
On-Demand Routing Geographic+On-Demand Routing
On-Demand Routing versus Geographic and On-Demand Routing
SP Implementation using SMSP Implementation using SM
SP application -> SMSP application -> SM Embedded system properties -> TagsEmbedded system properties -> Tags SM self-routing SM self-routing
Content-based routing using tagsContent-based routing using tags Geographical routingGeographical routing
Reference consistencyReference consistency unique tag created on the node which a spatial reference is unique tag created on the node which a spatial reference is
mapped tomapped to MappingTable (MT) maps spatial reference to the unique MappingTable (MT) maps spatial reference to the unique
tag and expected locationtag and expected location MT is carried by the SM that implements the SP applicationMT is carried by the SM that implements the SP application
Spatial reference access -> an SM migration to the Spatial reference access -> an SM migration to the node where the spatial reference is mapped tonode where the spatial reference is mapped to
Max_temp = {Left_Hill:Hot[1], timeout}.temp;
SP using SMs: ExampleSP using SMs: Example
Mobile sprinkler with temperature sensors
Left Hill Right Hill
Hot spot
Spatial Reference Access
Smart Message
ret = migrate_geo(location, timeout);if ret == LocationUnreachable ret = migrate_tag(yU78GH5, timeout);if (ret == OK) && (location == Left_Hill) return readTag(temp);else throw TimeoutException
Code Size breakdown for SM Code Size breakdown for SM (Application + SP Library)(Application + SP Library)
Code Size breakdown for Code Size breakdown for SP librarySP library
Testbed: 10 HP iPAQs with 802.11 cards and GPS devicesTestbed: 10 HP iPAQs with 802.11 cards and GPS devices
user nodeuser nodelight sensorlight sensorcamera nodecamera node
regular noderegular node
monitoredmonitoredspacespace
Execution Time BreakdownExecution Time Breakdown
Talk at a GlanceTalk at a Glance
Outdoor Programming ModelOutdoor Programming Model
Outdoor Execution ModelOutdoor Execution Model
ApplicationsApplications
Intelligent Distributed Intelligent Distributed Transportation SystemsTransportation Systems
Two projects:Two projects: EZCab: An Automatic EZCab: An Automatic
System for Booking CabsSystem for Booking Cabs Traffic Viewer: A Scalable Traffic Viewer: A Scalable
Traffic Monitoring SystemTraffic Monitoring System
Infrastructure-free approach based on car-to-car communication
short-range wireless communication location information from GPS receiver vehicle’s data from sensors through on-board diagnostic system (OBD-II)
EZCab IdeaEZCab Idea allow people to book cabs in dense cab allow people to book cabs in dense cab
area area using ad-hoc networking using ad-hoc networking
Finding Free CabsFinding Free Cabs
Booking a cabBooking a cab
Phase 2 & 3: Report and Phase 2 & 3: Report and ValidationValidation
TrafficViewTrafficView
• Enable drivers to monitor traffic in front of their cars, farther Enable drivers to monitor traffic in front of their cars, farther than they can seethan they can see
• Based exclusively on vehicle-to-vehicle ad hoc communicationBased exclusively on vehicle-to-vehicle ad hoc communication
How TrafficView WorksHow TrafficView Works
Receive data fromremote vehicle
Non-validateddataset
Validate
Validateddataset
Local data
Display
Broadcast data
Data AggregationData Aggregation
How to aggregate data to collect monitor How to aggregate data to collect monitor
vehicles as far as possible with “acceptable” vehicles as far as possible with “acceptable”
accuracy lossaccuracy loss
Natural SolutionNatural Solution Aggregate data for vehicles that are close to Aggregate data for vehicles that are close to
each othereach other Perform more aggregation as distance increasesPerform more aggregation as distance increases
Smart Phones as Universal Smart Phones as Universal Personal ServersPersonal Servers
Mobile phones and PDAs converge into Smart Mobile phones and PDAs converge into Smart PhonesPhones ProgrammableProgrammable Dual-connectivity (BT & GPRS)Dual-connectivity (BT & GPRS)
How to build pervasive applications on them?How to build pervasive applications on them?
SummarySummary
Spatial Programming makes outdoor distributed Spatial Programming makes outdoor distributed
Cristian Borcea (currently on the NJIT faculty)Cristian Borcea (currently on the NJIT faculty) Porlin KangPorlin Kang Nishkam RaviNishkam Ravi Pravin ShankarPravin Shankar Peng ZhouPeng Zhou Deepa IyerDeepa Iyer Akhilesh SaxenaAkhilesh Saxena
VisitorsVisitors Chalermek Intanagonwiwat (post-doc, currently on the Chalermek Intanagonwiwat (post-doc, currently on the
faculty of Chulalongkorn University, Thailand)faculty of Chulalongkorn University, Thailand) Oriana Riva (grad student, University of Helsinki)Oriana Riva (grad student, University of Helsinki) Tamer Nadeem (grad student, UMD)Tamer Nadeem (grad student, UMD) Marios Dikaiakos (faculty, Univ of Cyprus)Marios Dikaiakos (faculty, Univ of Cyprus)
Undergraduate students:Undergraduate students: Niket Desai, Peter Stern, Yan NuriyevNiket Desai, Peter Stern, Yan Nuriyev