October 14, 09 Saarland University Ubicomp A. Krüger 1 Ubiquitous Computing Winter term 2009 Prof. Dr. Antonio Krüger
October 14, 09 Saarland University Ubicomp A. Krüger 1
Ubiquitous Computing
Winter term 2009
Prof. Dr. Antonio Krüger
10 December 2009 Ubicomp Antonio Krüger Winter term 20092
Topics Today
Context Awareness• Definitions• Example projects• The Context Toolkit• Low level context acquisition• Some implementation concepts
10 December 2009 Ubicomp Antonio Krüger Winter term 20093
Trying to define context
! Context acc. to Bill Schilit (Active badge 1992 + Parctab 1994)• „mobile distributed Computing“: Mobile
Computers and Users• The three (spatial) Ws:
- Where is the user?- Who is near the user?- What services can the user use in the vicinity?
• Important: the temporal changes of the 3 Ws
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger4
Trying to define context (2)
! Context properties• Location• Lighting/brightness• Sound/noise• Network availability• Network bandwidth• Group constellation
• Dialogue history• Position history• User interest• Time of day• Orientation • Speed • Temperature
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger5
Example project: Active BadgesOlivetti / AT&T, Schilit, Hopper, Harter, et al.
! Teleport• Redirect screen output
from "home" computer to nearby computer
! Phone forwarding• Automatically forward
phone calls to nearest phone
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger6
Active Badges: Technology! Badges emit infrared (IR remote)
signals! 1 signal every 15 sec.! Avoid 2 badges in sync
• use high tolerance components• Light sensor changes interval• " switched off when in the dark
! Button to trigger events! Sensors distributed in the building! Central server scans regularly for
„badge sightings“! Over 1500 badges and 2000
sensors used worldwide
“A disadvantage of an infrequent signal from the badge is that the location of a badge is only known,at best, to a 15-second time window. However, because in general a person tends to move relativelyslowly in an office building, the information the Active Badge system provides is very accurate.” ;-))
10 December 2009 Ubicomp Antonio Krüger Winter term 20097
Active Badges: Initial Services! FIND (name)
• Provides the current location of the named badge and, if it has recently moved, a list of all the locations it has been sighted at in the last five minutes along with the likelihood of finding it at each.
! WITH (name)• Locates a named badge and provides information about other badges that are in
the immediate locality of that badge.! LOOK (location)
• Allows an investigation to be made of the badges that are currently near the specified location.
! NOTIFY (name)• An alarm mechanism that generates an audible indication of when the named
badge is next sighted after executing the command. ‘NOTIFY’ is particularly useful when trying to deliver an urgent message to a member of staff who is out of the office on business for long periods of time.
! HISTORY (name)• Generates a condensed report of the location history for the named badge during
a one-hour period. The system intentionally does not record any location data on a permanent storage medium, to dispel concern about long-term monitoring of an employee’s movements.
10 December 2009 Ubicomp Antonio Krüger Winter term 2009
Active Badge Video (Hopper et al CHI 93)
8
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger9
Xerox ParcTabhttp://sandbox.parc.xerox.com/parctab/
! Infrared network• Base stations in the ceiling• Low bandwith, modulated carrier• Transmission radius ~7m
! Mobile tab-sized devices• Unistroke input via pen
! Context-aware applications:• Information access• Communication• Collaboration
10 December 2009 Ubicomp Antonio Krüger Winter term 200910
Information Access
! Weather (Internet/local)! Dictionary, Thesaurus ! UNIX file browser ! WWW browser (mit Einschränkungen)! Calendar manager (Sun's cm) ! Dateimanager (ortsabhängig)
10 December 2009 Ubicomp Antonio Krüger Winter term 200911
Communication
! email: permanent access! pager! locator! „Communicator“, media-space controller
• Tab proposes best communication devices in the surroundings and initiates connection
10 December 2009 Ubicomp Antonio Krüger Winter term 200912
Collaboration
! Tab as pointing device • Remote pointer control for liveboard• Move pointer with pen on the tab screen
! Tabdraw: collaborative drawing• One canvas per room
! Arbitron (Tool for voting)• Tell the presenter to speed up or slow down
10 December 2009 Ubicomp Antonio Krüger Winter term 200913
Other Applications
! Remote Control• Control of physical environment• Universal (self-configuring) remote control
! Local (on Tab) applications• For offline use• Note pad for memos
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger14
Techniques in Context-Aware Computing (Schilit et al. 1994)
! Proximate selection• E.g., list closes devices first
! Automatic contextual reconfiguration• E.g., automatically select
nearby devices for output! Contextual information &
commands• Commands with different
meanings in different contexts! Context-triggered actions
10 December 2009 Ubicomp Antonio Krüger Winter term 200915
Context-triggered actions! Simple If-Then Rules, similar to Unix CRON-
Demon:• Coffee Kitchen arriving „play -v 50 /sounds/rooster.au• schilit * attention „emacs -display $NEARESTHOST:0.0“
! Contextual reminders: information is displayed under certain conditions. Example:
$DATE=„after April 15“ AND $TIME=„after 10“ AND $room=„35-2-200“ AND $WITH-USER=Adams“AND Color($DISPLAY)=„true“
10 December 2009 Ubicomp Antonio Krüger Winter term 200916
Context-sensitive Systems
! Scientific problems• How to recognize relevant context (Sensors)• How to use the obtained context information
(adaptive/reactive Systems)- Implicit vs. Explicit control of systems- Reactive Systems- Situated Systems- Adaptive Systems
10 December 2009 Ubicomp Antonio Krüger Winter term 200917
Situated Systems
! Activity as context• Location and identity
- Simple, but only weak characterization of situations• Complex Sensors (e.g., cameras)
- rich information- high computing power needed- endangered privacy
• Integration of many simple sensors- TEA Project Karlsruhe - Context Toolkit
10 December 2009 Ubicomp Antonio Krüger Winter term 200918
TEA ProjectEU-funded 1998-2000, TECO, Starlab/Be, Nokia/Fin
! TEA: Technologies for Enabling Awareness• Combination of simple sensors instead of
complex image processing• Combine multiple sensors with context
information • HW/SW-Addon for mobile devices
- low-energy, low-cost- Target platform: cellular phone
10 December 2009 Ubicomp Antonio Krüger Winter term 200919
TEA Project! GSM Telephone as the TEA Host
• User expects different behavior of the phone according to the situation
• State of the art: manual profile selection• TEA enables automated profile selection
depending on sensor data! Application Context-Call
• Caller dials the number of the user, and is told the user’s situation by TEA (e.g., „In a meeting“)
10 December 2009 Ubicomp Antonio Krüger Winter term 200920
TEA Project! TEA-Hardware:
• light, audio, acceleration, and temperature sensors• Microcontroller controls the sensors and extracts hints• Final decision is made on the host
First prototype second Prototype
10 December 2009 Ubicomp Antonio Krüger Winter term 200921
Active artifacts
! Concept:• Determine activity where it occurs• Add “self perception” to everyday things• Communicate their own state• The artifact digitally “supports” its own
applications• Example: MediaCup
- http://mediacup.teco.edu/
10 December 2009 Ubicomp Antonio Krüger Winter term 200922
Mediacup (Teco, Univ. Karlsruhe)
! First experimental „active artifact“! Technical Info:
• PIC-Microcontroller, 15k/384Byte, low-energy• IrDA physical level communication• 3 acceleration, 1 weight, 1 temperature sensor• 2 condensers as power supply
! „Self perception“:• Reading out sensors periodically• Compute important events : in the shelf, full, empty,
currently in use, etc...
10 December 2009 Ubicomp Antonio Krüger Winter term 200923
Mediacup! Small number of cups
! In use since 1999
! 95% correct recognition of Multi sensor events
! Important design criterion: Energy consumption, heavy influence on outward appearance
10 December 2009 Ubicomp Antonio Krüger Winter term 200924
Context acquisition through Cooperation! Cooperating Arteficts
(Strohbach et al 2004)
! Knowledge is exchanged between artefacts that sense their local context
! Example: chemical containers
10 December 2009 Ubicomp Antonio Krüger Winter term 200925
Each Artifact is context sensitive
10 December 2009 Ubicomp Antonio Krüger Winter term 200926
Sharing knowledge between artifacts
10 December 2009 Ubicomp Antonio Krüger Winter term 200927
Fact base
10 December 2009 Ubicomp Antonio Krüger Winter term 200928
Inference rules
10 December 2009 Ubicomp Antonio Krüger Winter term 200929
Hazard Examples
Hazard critical mass!Hazard incompatible
10 December 2009 Ubicomp Antonio Krüger Winter term 200930
Limited Resources
! Technical resources (of the environment)• Available media: e.g. displays, loudspeakers• Media attributes: screen size & resolution,
colors• Quality of positional data: user’s location (e.g.
indoor/outdoor), orientation and speed• Available CPU-power and memory• Communication bandwidth
10 December 2009 Ubicomp Antonio Krüger Winter term 200931
Limited Resources (2)
! Cognitive Resources (of the user)• Cognitive load:
- Use of working memory- Time pressure
• Familiarity with the environment• Personal preferences:
- Media, content and presentation styles• Limited vision, hearing, motor skills, etc.• Communication abilities: limited use of
modalities, e.g. use of gesture and speech
10 December 2009 Ubicomp Antonio Krüger Winter term 200932
Types of resource adaptivity
! Adapt to available resources• Technical resources• Cognitive resources
! Resource-adapted Systems• Systems which are optimized towards a certain
resource limitation• Result is optimized to the limitations of the sensors in
the environment.• Different resource situations lead to failure• Examples: TEA, MediaCup
10 December 2009 Ubicomp Antonio Krüger Winter term 200933
Types of resource adaptivity
! Resource-adaptive systems• Implementation of a strategy to adapt to a limited
resource• Result is improved with increasing availability of
resource (example: any-time algorithm)
! Resource-adapting systems• A Meta-strategy chooses between different resource-
adaptive strategies to fit the resource limitations• Implementation of a meta-cognitive system level
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger34
Context Toolkit (with slides courtesy of Anind Dey)
! Anind K. Dey (CMU Pittsburgh)! Toolkit to support Context-Aware
applications! Strong formalization of “context”! Implementation in Java. ! Can be distributed on several machines
in the environment
10 December 2009 Ubicomp Antonio Krüger Winter term 200935
Context and Context-Awareness
! Focused on input
! Context: any information relevant to an interaction that can be used to characterize the situation of an entity
! Context-Awareness • General model of interactive computing• Addresses subset of ubicomp problems: input
10 December 2009 Ubicomp Antonio Krüger Winter term 200936
Value of Context
! Potential for improved usability• Very important for mobile users with poor input
devices
! “Smarter” applications
10 December 2009 Ubicomp Antonio Krüger Winter term 200937
Example
! Tour guides, travel assistants, personalization software
! Reminder to buy milk• When to deliver: not time/location specific• How to deliver: appropriate modality
10 December 2009 Ubicomp Antonio Krüger Winter term 200938
Building Applications
! M. Weiser: The whole point of ubiquitous computing, of course, is the applications.
• But … what if the applications are hard to build? And, what if this inhibits our ability to build compelling applications?
10 December 2009 Ubicomp Antonio Krüger Winter term 200939
Why Context is Hard to Use
! Acquired from sensors• Not just keyboards and mice – lots of heterogeneous
devices
! Need to abstract data
! Distributed
! Dynamic
10 December 2009 Ubicomp Antonio Krüger Winter term 200940
Results of Difficulties
! Ad hoc application building• Difficult to build, reuse and evolve
! Small variety of sensors! Small variety of context: mostly location ! Few applications, mostly simple: mostly
presenting context
! Practical: difficult to prototype, test and evaluate
10 December 2009 Ubicomp Antonio Krüger Winter term 200941
Toolkit Requirements
! Context specification• Discovery
• Storage• Separation of concerns
• Interpretation• Transparent communications• Constant availability
10 December 2009 Ubicomp Antonio Krüger Winter term 200942
Look to input handling ! Graphical User Interface (GUI) widgets
• separation of concerns• callbacks and attributes• query/subscribe• common interface
• e.g. button
10 December 2009 Ubicomp Antonio Krüger Winter term 200943
Context Widgets
! Responsible for acquiring and abstracting data from particular sensor, separation of concerns, storage
Widget
Sensor
Widget
Application Application
Sensor
ContextArchitecture Face
Recognition
Location Location
In/Out Board
Smart Card Reader
10 December 2009 Ubicomp Antonio Krüger Winter term 200944
Context Interpreters
! Convert or interpret context to higher level information
! Context not available at appropriate level
Face Recognition
Location Location
In/Out Board
Smart Card Reader
ID to NameInterpreter
10 December 2009 Ubicomp Antonio Krüger Winter term 200945
Context Aggregators
! Collect context relevant to particular entities (recall definition)
! Further separation, simplifies design
Face Recognition
Location Location
In/Out Board
Smart Card Reader
ID to NameInterpreter
BuildingAggregator
10 December 2009 Ubicomp Antonio Krüger Winter term 200946
Context Toolkit Framework! Supports real-world model/methodology and provides library
(distributed: XML/HTTP, input-focused)! Component model: facilitates building of applications
Widget
Sensor
Widget
Application Application
Interpreter InterpreterAggregator
Sensor
ContextArchitecture
Service
Actuator
10 December 2009 Ubicomp Antonio Krüger Winter term 200947
Experiences: Benefits
! Provides separation of concerns
! Lightweight integration and re-use of components
! Easy to create and evolve apps, allowing exploration of the design space• Add context to context-less apps• Add more context to context-aware apps
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger48
Aware Home (MANSE ’99)
! Great testbed for context-aware computing! 3 goals: elderly, infants, everyone! Context Toolkit is the s/w infrastructure in the
Aware Home
10 December 2009 Ubicomp Antonio Krüger Winter term 200949
Applications Built
! Simple use of location:• Turn lights on and off (perform service)
! Location and id (perform service)• Information Guide: present info about user’s
group (CHI ’99)
• Context-Aware Mailing List
10 December 2009 Ubicomp Antonio Krüger Winter term 200950
In/Out Board – 3 versions (CHI ’99)
! Context used: location, identity, time! How used: present context
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger51
In/Out Board Architecture
LocationWidget
In/Out Board
iButtonDock
ID to NameInterpreter
Web-basedIn/Out Board
Context-AwareMailing
List
PinPoint3D-iD
• Simple app demonstrates support for reusability (don’t have to re-build infrastructure on per-application basis) and evolving applications
10 December 2009 Ubicomp Antonio Krüger Winter term 200952
Serendipitous Meetings• Context used: location, id, time, activity • How used: present, perform service, tag• record and tag drawings and audio for later retrieval
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger53
Playbackcontrols
Filters
Ink written beforecurrent time is in original color
Ink written aftercurrent time is in lighter color
Current time within sessionSelected session
Selected dayDay containing whiteboard activity
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger54
Meeting Architecture
ContextArchitecture
For each possible location of the
mobile board
LocationWidget
DUMMBO
LocationWidget
ID to NameInterpreter
iButtonDock
iButtonDock
10 December 2009 Ubicomp Antonio Krüger Winter term 200955
Conference Assistant (ISWC ’99)
! Context used: location, multiple levels of identity, activity, time
! How used: present, service, tag
Slide
User Notes
Interest Control
Audio/VideoIndicator
Slide text User notes
Retrievedslide
Query Interface
Schedule
context widgetsIdentity, Location, Activity
Joe Smith context
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger56
Conference Assistant Arch.
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger57
Low level context recognition(Cakmakci et al. 2002)
! Design “context aware hardware”
! Enhance wearable computing
! Detect simple user activities, like sitting, walking, looking at the watch
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger58
Low level context recognition! Use statistical modeling techniques from robotics
to determine context! Use accelerometers to record movement changes! Apply Bayes rule to determine probability of certain
contexts:
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger59
Low level context recognition! First simple Experiment: Detect whether users
are sitting, standing and walking
Acceleration over 4500 data points during the experimentUse the first 1000 data points for learning
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger60
Low level context recognition
! Results of recognition
10 December 2009 Ubicomp Antonio Krüger Winter term 200961
Low level context recognition
! 2nd experiment: Detect when users glance at their watch
! Idea: reduce power of watch (e.g. toggle display) during use
! Extremely important for a wristwatch computer (IBM WWC: ARM7 processor, 8Mb flash memory, 8Mb of DRAM, serial, IRDA, and expansion interfaces).
10 December 2009 Ubicomp Antonio Krüger Winter term 200962
Low level context recognition
Evaluation set in a blind experiment
10 December 2009 Ubicomp Antonio Krüger Winter term 200963
10 December 2009 Ubicomp Antonio Krüger Winter term 200964
Low level context recognition
Use a single hypothesis approach to model “wrist watching”(Only one well defined class)
10 December 2009 Ubicomp Antonio Krüger Winter term 200965
Low level context recognition
Results:Probability of certain data points belonging to a wrist watch action
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger66
Intelligent adaptive assistants
indoor
outdoor
Two main objectives:
• Generate adapted route descriptions• Generate adapted presentations
Museum
Exhibit
Virtual Window
PDA
REAL:Resourceadaptive Navigation
PEACH: Personal Experiences in Active Cultural Heritage
10 December 2009 Ubicomp Antonio Krüger Winter term 200967
Research Problems
! Given an instrumented environment and a user’s task• How to adapt route descriptions to
- the technical resources of the environment?- the cognitive resources of the user?
• How to adapt presentations to - technical resources of the environment?- user preferences and interests?
10 December 2009 Ubicomp Antonio Krüger Winter term 200968
Approach
technical resources X cognitive resources X user’s task
?
Reaction of the intelligent environment• Content presentation• Interaction possibilities
A2: Use a declarative approach to specify system behavior under limited resources, represent what has to be presented!
A1: Design an architecture that supports resource adaptivity at several levels.
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger69
SpatialKnowledge
RouteKnowledge
SurveyKnowledge
LandmarkKnowledge
Segment
Task
SegmentTask
Directions
ObjectDescription
AdditionalInformation
SegmentPresentation
SupportTechniqueTechnique
Abstraction
SketchGeneration
MetaGraphics
PresentationAgent
CameraTechnique Shot
Zoom
Pan
Movement
World InMiniature
AnimatedWalkthrough
MultipleViews
GraphicalPresentation
ResourceRestriction
CognitiveRestriction
PositionResolution
OrientationResolution
TechnicalRestriction
DeviceRestriction
GraphicCapabilities
Bandwidth
CognitiveLoad
TimeConstraints
Comm.Abilities
WorkingMemory
SpatialFamiliarity
SystemKnowledge
DesignKnowledge
DomainOntology
UserModel
3DModel
10 December 2009 Ubicomp Antonio Krüger Winter term 200970
Adaptive graphical route descriptions
Resource sensitiveroute
Wayfinding
Presentation script
Presentation PlanningLimited Resources- technical- cognitive
Media specificInformation
Presentation
Adaptive routedescription
10 December 2009 Ubicomp Antonio Krüger Winter term 2009
Hier weiter
71
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger72
Resource-adaptive route finding
Optimization criteria:
• Shortest (fastest) route • Reduce amount of turning points, • Reduce complexity of turning points• Exploit spatial familiarity
A
B
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger73
Planning Formalism
Show-way-description
sketch-draw-arrow
sketch-show-start-point
sketch-draw-circle sketch-draw-cross
sketch-show-end-pointsketch-show-trajectory
sketch-show-trajectory-step
sketch-show-trajectory-segment sketch-show-decision-point
additional
show-sketch show-3d-walkthrough
alternative
sketch-show-path sketch-show-ground-plan
incremental
sketch-draw-thumbnail sketch-write-label
conditional
10 December 2009 Ubicomp Antonio Krüger Winter term 200974
Representing the domain ontology
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger75
A Hybrid Navigation System
Presentation
Server
Information Booth
Mobile Indoor
Mobile in/outdoor
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger76
Information Kiosk: Techniques
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger77
Camera Styles
Egoview
Front and shoulder camera
Birdʻseye view
• Position
• Rotation• Viewing angle
10 December 2009 Ubicomp Antonio Krüger Winter term 200978
Information Kiosk:Presentation Agent
10 December 2009 IfGi Universität Münster Ubicomp A. Krüger79
Graphical Techniques and Types
bad -- - 0 + ++ good
Our approach: hybrid representation of spatial knowledge • Geographic coordinates
– Geometrically precise– Data acquisition– Visualization (mainly for outdoor purposes)– Example: WGS84 coordinates
• Symbolic location identifier– Geometrically abstract/vague– Used to describe situations– Visualization (mainly indoors)– Example: model of a spatial ontology
Related Work• Hybrid Models [Leonhardt 98]
– Combines symbolic and geometrical location descr.• Hierarchical Location Identifier [Jiang02]
– Combines symbolic and geometrical location descr. in a single identifier
• Nexus [Dürr03]– Symbolic object-graph for geo-broadcasts
• Problem: location description is not embeded into the situational context of the user:– No relation to user prefrences and goals– Reduced adaptation of the interaction
User Modeling and Spatial representations (Heckmann 2005)
UbisWorld ( Ontology + Instances )
Physical Ontology
Spatial Ontology
Temporal Ontology
User/Group
Activity Ontology
Classes & Predicates
Device/Object Location
Individuals (id, label, category, parents)
Time 3 47
2 13 4 5
Relations
Situation Ontology
28
OWLDamlOILRDFSQL
Levels of description Instance of concept “room”
Building 36
Info !
Campus
Info !
Building 45
Info !
Building 28
Info !
Room 124
Info !
Hallway
Info !
Kitchen
Info !
2nd Floor
Info !
URI=//www.u2m.org/#b36
URI=//www.cs.uni-sb.de/#b36Room124
Geometric_model
Geometric_model
Symbolic-model
Symbolic-model
Refinement
Parent
Spatial Granularity Level: Street - Building
Map of University Campus
Map of 2nd Floor
Spatial Granularity Level: Floor - Room
URI=//www.uni-sb.de/#CampusMap
URI=//www.cs.uni-sb.de/#b362ndFloor
N
0
Infrared beacons
RFID-beacons
Mapeditor Yamamoto
Permeable edges
Selected polygon
Polygons representing rooms
Symbolic identifier
Mobile Multimodal Interaction (M3I) Plattform• Improved interaction with small screen devices
– Navigation instructions and spatial exploration• Mobile 2D- und 3D-Graphics • Embedded speech recogntion and synthesis
– Commands, Route information, Landmarks• Gesture recognition• Embedded into an infrastructure
– Positioning– Routing– Local maps
• Universal platform– Special target groups (e.g. visually impaired users)– Enables Empirical evaluations in the field
Implementations-plattform:Pocket PC
Mobile Graphics
2D-Graphics 3D-Graphics Ego-perspectiveTop-down views
Mobile gesture recognitionTypes• Intra – gesture on mobile screen• Extra – gesture in the world
Intra-gesture library• points (buildings)• lines (streets)• circles (areas)
Sensors:
Intra gestures: Point and slide Extra gesture
Magn.
Speech recognition and synthesis
„Command and Control“ Landmarks, point gesture Street, line gesture
Fusion with Intra-gesture
Example M3I
Positioning technology in M3I• Local sensor-data collection and remote
processing– RFID– infrared– GPS– Self localization (Kray 2004)
• Determining the position – Mobil
• Advantage: no infrastructure needed, robust– infrastrucutre
• Advantage: use of external services possible
IRRFIDRFID-Leser
Indoor positioning
RFID-beacon
Infrared-beacon
Strong Signal Str.
Weak Signal Str.
Route
Actual position
Receive ID of IR-beacon
A hybrid method for indoor localisation
Activate RFID-beacons in the vicinity
9
A hybrid method for indoor localisation
1035
1036
1034
Receive responses of RFID-tags (ID and signal strength)
1039
A hybrid method for indoor localisation
Spatial Audio (2 von 8)
Plasma-screen(touch sensitive)
RFID shelf
Projector-Camera Unit
Products with RFID-tags
IR Beacons
Optical Markers
REAL: Instrumented Environment S.U.P.I.E.
REAL architecture: Navigation
Communication using Blackboard Infrastructure
PocketPC
GISGeometric Location
Model
Local Mapand Route
UbisWorld• Locations• Persons• Objects• Activities
IRRF
GPS
PedestrianNavigationAssistant
Shopping Cart
RF Antenna
TabletPC
SpeechStylus
Shop AssistantPositioning bySensor Fusion
Location
ID
Routing
PresentationPlanner
SpeechStylus
Speech + Gesture
Product ID
Presentation
Plasma Display
RFID Shelf
RF Antenna
Request
Smart Door Displays
Infrared BeaconRF Beacon
Productwith RFID Tag
ID CardCybercode
ID CardRFID Tag
Smart DoorDisplays
Beamover
Touchscreen
Spatial Audio
Search Light
Pedestrian Navigation Assistant
Presentation
PlansKno
wle
dge
Personal Public
Ser
vice
Sen
sing
Use
r Int
erac
tion
Pos
ition
ing
M3I Platform Assistance through instrumented space
Navigation in instrumented spaces
Smart Door DisplaysMixed reality (projection)Mobile client
Indoor Navigation with Smart Door Displays
Indoor Navigation with Smart Door Displays
Example
Multi-user navigation w. public displays(ruzkio,schmidt,krüger, CHI 2005)
• Synchronized signals on personal and public devices
• Vibration alarm of mobile phones as a specialization of a one-bit display
Initial User study
• Lab setting with 14 participants• “Rotating compass” projected on the floor• In 140/140 trials participants identified the right direction• Two modes: “phone in hand”, “phone in pocket”• Problem: What if more than one display is in sight?• Problem: What if the phone is carried in a bag.
Guiding attention through a virtual room inhabitant(micro-navigation)
Using a geometric model of the room for a distortion free projection
Speech dialogues with objects in the environment(Talking Objects Paradigm)
Embedded speech recognition. Deictic reference triggered by physical action.
Comparison shopping triggered by multimodal interaction
Fusion of intra-, extra-gesture and speech input
Experiment:age-dependent navigation • Investigation of the sensomotoric-cognitive modules related to
navigation• Treadmill-paradigm. Visit in a museum• 32 younger und 32 older Subjects• Varying topologies and navigation aids• Dependent Variables: measured walking distance to target,
landmark knowledge
Lab setting „Old Town“ „City Block“
Results
•Huge gender effect. Possible explanation: Women rely on different representation of space while navigating
•Huge age-effect•Reduction of the senso-motoric load helps older man to improve navigation
Incidental route learning • How does assisted navigation influence learning of our
spatial environment• Which modality should be used to encode navigational
instructions?– Gender differences?– Incremental vs. complete?
• Which technology should be preferred:– PDA-Display– Clip-on
• Dependent Variables:– Route reconstruction– Landmark knowledge
Incidental route learning• Field setting in the Zoo of Saarbrücken• 32 VP (16 m, 16 w)• Wizard of Oz Experiment, Navigational
instructions triggered through a second PDA
Incidental route learningPresentations:
„Hier links“„…“Dependent variables:
Route reconstruction Map reconstruction
Conclusions• Hybrid knowledge representation important
– Embedded into a representation of the userʼs situation• Multi-modal interaction with mobile devices
– Speech and Gesture improve robustness of interaction
• Taking advantage of instrumentation helps to navigate in indoor settings
• Empirical evaluations are necessary to improve spatial appliances (lab and field)
Future work• Improve detection of situations
– Technical constraints– Cognitive constraints (e.g. Bio-sensors)
• Develop framework for instrumented indoor navigation– Mixed usage of private and public devices
• Continue to develop M3I towards a platform to assist spatial cognition (e.g. for the elderly).
Thank you very much
Further information available at:
http://w5.cs.uni-sb.de/real