Page 1
Grounding to the Real WorldGrounding to the Real World-- Architecture for Ubiquitous Architecture for Ubiquitous
Computing Computing --ISMIS 2003ISMIS 2003
Hideyuki NakashimaHideyuki Nakashima中島秀之中島秀之
Cyber Assist Research Center, AISTCyber Assist Research Center, AIST
Page 2
[email protected] 2
Background
• Information technologies are advanced to the point where it becomes feasible to assist ordinary people enjoy the full capability of information processing powers.– Although the Internet opened up huge
possibilities for daily use, users still have to learn a lot of special concept to use it.
Page 3
[email protected] 3
Background 2
• With the help of various sensing technologies, it is time to design arich, keyboard-less interface and provide an ubiquitous computing environment for ordinary people in everyday life. Ubiquitous agent technology plays essential role here.
Page 4
IntroductionIntroduction
Page 5
[email protected] 5
TICKET
Take the next train.Change to A line at Ueno station.The fare is YY yen.
I want to go to Maebashi station.
Good Old DaysGood Old Days
Page 6
[email protected] 6
券売機券売機 券売機
●●●●
××××
△△△
○○○○○
▲▲▲▲
120
560
340
780
900
□□□ 210
※※※※ 400
※●※●※
×○×
△■△■
○□○□
▲×▲×▲
120
560
340
780
900
□◎□◎ 210
■※■ 400
●●●●
××××
△△△
○○○○○
▲▲▲▲
120
560
340
780
900
□□□ 210
※※※※ 400
※●※●※
×○×
△■△■
○□○□
▲×▲×▲
120
560
340
780
900
□◎□◎ 210
■※■ 400
券売機
・・・・・
When she getsolder...
When she getsolder...
Page 7
[email protected] 7
券売機 券売機 券売機 券売機
●●●●
××××
△△△
○○○○○
▲▲▲▲
120
560
340
780
900
□□□ 210
※※※※ 400
※●※●※
×○×
△■△■
○□○□
▲×▲×▲
120
560
340
780
900
□◎□◎ 210
■※■ 400
●●●●
××××
△△△
○○○○○
▲▲▲▲
120
560
340
780
900
□□□ 210
※※※※ 400
※●※●※
×○×
△■△■
○□○□
▲×▲×▲
120
560
340
780
900
□◎□◎ 210
■※■ 400
How much does it cost?
What is the best route to Maebashistation?
Page 8
[email protected] 8
券売機 券売機 券売機 券売機
●●●●
××××
△△△
○○○○○
▲▲▲▲
120
560
340
780
900
□□□ 210
※※※※ 400
※●※●※
×○×
△■△■
○□○□
▲×▲×▲
120
560
340
780
900
□◎□◎ 210
■※■ 400
●●●●
××××
△△△
○○○○○
▲▲▲▲
120
560
340
780
900
□□□ 210
※※※※ 400
※●※●※
×○×
△■△■
○□○□
▲×▲×▲
120
560
340
780
900
□◎□◎ 210
■※■ 400
あっ!
Page 9
[email protected] 9
Please go through..
The next train will leave from platformNo 2 to your right.
TURN RIGHT TURN RIGHT
speaker
Connection to Internet
Time table server
Seat reservation system
My-button
LED display
Station master computer
Station LAN
TokyoRF tag
Unawareable environmentUnawareable environmentIR communicator
Position sensor
Page 10
[email protected] 10
Ubiquitous Computing• Omnipresence of computing power
• Human centered services– The opposite direction from virtual reality where the
reality is reconstructed in the digital world
– Computers come out to the real world
• Gadgets are hidden– calm computing– disappearing computer
– oxygen
– unawarable computers
Page 11
[email protected] 11
MIT Oxygen: Intelligent Space• Context aware environment: computation “in the air”
Stereo Cameras: to track persons and objects
Stereo Cameras: to track persons and objects
Active Cameras: for video-conferencing and meeting archive support
Active Cameras: for video-conferencing and meeting archive support
Microphone Arrays: beam-forming for untethered spoken interface
Microphone Arrays: beam-forming for untethered spoken interface
Page 12
[email protected] 12
Before:People had to learn to use machines
Computers in the Digital WorldComputers in the Digital World
People in the Real WorldPeople in the Real World
Internet schoolInternet schoolPC schoolPC school
Page 13
[email protected] 13
After:Grounding is the key technology
Digital WorldDigital World
Real WorldReal World
groundinggrounding
Page 14
[email protected] 14
“Grounding”
• Connection between digital data and real things– similar to “symbol grounding problem”
– (but not the same)
• Total grounding is not necessary nor possible– utilization of artificial means or human aid
Page 15
[email protected] 15
From WWW to HNM
• Wherever, Whenever, Whoever
→ Here, Now, Me (situatedness)
• At the same time:
Just as much as you want to know
→ intelligent information filter
• And,
Just as much as you want to be known
→ privacy
Page 16
Location based communicationLocation based communication
Page 17
[email protected] 17
Human Centered Infrastructure
ID-based communication
ID-based communication
Location-based communication
Location-based communication
Broadcasting satellite36000km~780km
Broadcasting satellite36000km~780km
High-altitude aeroplane/ship20km
High-altitude aeroplane/ship20km
broadcastbroadcast
broadcastbroadcast
Sensor networkSensor network
Page 18
[email protected] 18
Communication Methods
Local 1-10 meters global
Magnetic cardMagnetic cardICIC--cardcard
NetworksNetworksPhonesPhones
BroadbandHigh-speed
LowLow--powerpowerShortShort--rangerange
LocationLocation--basedbased
IDID--basedbased
Page 19
[email protected] 19
Advantages of Location Based Communication
<x, y, z, t>
• Location is one of the essential keys for grounding the digital (logical) world to the real world.– Situated information processing and services
• Interaction without ID– Possibility of PRIVACY protection
– For better human interface
– Application to emergency situations
Page 20
[email protected] 20
Various location-based communication(addressing) methods
(cricket)*(mobile phones)*
(cricket)*(mobile phones)* carrier pigeoncarrier pigeon
Logical addressing“passenger at 12C”
Positioning by server
Self-Positioning
Physical addressing“you”
(bluetooth)*DSRC
(bluetooth)*DSRC CoBITCoBIT
*positioning only
Page 21
[email protected] 21
Security vs. Privacy vs. Convenience
securitysecurity privacyprivacy
convenienceconvenienceThey basically contradict each other with some exceptions
Page 22
[email protected] 22
Needs technology for privacy protection
amountofservice
privacy protection
not zero
Page 23
[email protected] 23
Dynamic/Voluntary Control of the Balancing Point is Necessary
• Trade-off between the amount of services and the amount of personal information
• Let the user set the point!
Privacy Security
Page 24
AgentAgent--based Network based Network ArchitectureArchitecture
Page 25
[email protected] 25
Ubiquitous Computing
How do we access and control devices
Intuitive User Interfacethat uses real world propertiessuch as location(point and click)
Page 26
[email protected] 26
Why Agents?
• Aids for human cognition– interface
• Resource control– cognitive resources
– devices• multiple users and• multiple devices
Page 27
[email protected] 27
Cognitive Resource Control
• SmartKom example:– cognitive resource of human
• when running terse• when walking• when standing verbose
Page 28
[email protected] 28
DFKI: SmartKom
• positioning– indoor: light beacon
– outdoor: GPS
Page 29
[email protected] 29
・Maintains who reserved which time slot・Controls remote access・The reserver has the access right
homeVCR
toilet
bathliving
air conditionerthermometer
thermometer
・The access right belongs to who reserved・Access right can be delegated・The device may act as a mediator
homeserver
RFID tagRFID tab
RFID tag
Son
父
×
×
○
!
town
OK Dad, go ahead...
Access Control of Home AppliancesBased on Location and Priority
My son has a reservation. Let’s ask him.
someone is using the air conditioner・change reservation
or, delegate authority
・Deny remote access when someone is near-by
・communication center for the family
Page 30
[email protected] 30
CONSORTS
InferenceEngines
Personal Agents
ServiceAgentsUser1 Usern
Communication and censoring
devices
Physical WorldContentServer
Devicewrapper
Agent communicationAPIphysical link
Spatial information database- sensor-based geometrical representation- ontology- location information
Spatio-Temporal Reasoning Agent
Camera, Wireless LANCoBIT, etc.
User modelmanager
Physical Grounding
Web Services
Page 31
[email protected] 31
Spatio-temporal Inference Agent in Consorts
spatio-temporal map(Segment Representation)
Inference engine:spatio-temporal relations and their changes (distance, phase, connection etc.)
device wrapper agents
personal agents,service agents
request, reply, inform
time
sensor data
Spatial informationdatabase
Page 32
[email protected] 32
interpretation
Sensor information at location (x,y) mapped onto a map
Geometric representation of sensor information
Museum
1st Floor
Entrance
3rd Floor2nd Floor
Room CRoom A Room B
Painting BPainting A Painting C
Hierarchical representation of physical space
Painting D Painting D
Page 33
[email protected] 33
Ubiquitous Computing with FIPA-Agents
• Direct connections to sensors are not good
• Agents mediate communication of sensor information
• Agentcities Network• http://www.agentcities.net/
– EU centered worldwide research network based on FIPA standard
– 134 platforms• EPFL, BT, Motorola, etc.
– Tokyo.Agentcities.Net (CARC)• http://www.carc.aist.go.jp/agentcities/
Page 34
[email protected] 34
A Snapshot of Our Museum Information Support System
Main Window (Museum Map)
Information Window
Message WindowMessage Window
Human Icon: Current Location
Yellow Area: Service Zones
Blue Line: Users’ Trajectories
Page 35
[email protected] 35
UBKit
UBKit: Software construction tool kits
middleware
infrared
SOAP DNS
network
IEEE802.11 BluetoothMicroservers
voice positioning
real world
CONSORTSCONSORTS
Page 36
[email protected] 36
Micro servers + Optical Communication forIntegrated Remote Control System
Ultra small low energy micro server(Linux) coin size
Self configuring plug and play network
radio
positioning system
voice recognition
sensor networkinfrared
Lightweight middleware (service directory, SOAP)
sensorgateway
UBKit (Ubiquity Building Toolkit)
Integrated control of all home applianceswith intuitive interface
Page 37
Connection toConnection tothe Semantic Webthe Semantic Web
Page 38
[email protected] 38
Annotation as Grounding
Content(Real World)
SemanticWeb
AnnotationAnnotation
Semantics forComputers
Sharing semanticsbetween computers
and human(Grounding)
Page 39
[email protected] 39
Multimodal Content Processing
• Grounding linguistic content to the physical world– annotation with location– interaction through CoBIT
• User modeling– record of movement– record of inquiries
• Summarization and interactive presentation
Page 40
Cyber Assist ProjectCyber Assist Project
Page 41
[email protected] 41
The goal:Grounding the digital world to the real worldfor “here, now, and to me” service
informationinformation
Digital worldDigital world
Real world
goodsgoods
humanhumanSemantic WebSemantic Web
Ubiquitous ComputingUbiquitous Computing
Page 42
[email protected] 42
My Button• One button
– Do what I want here, now.
– No button?Activation by
brain wave?
• The ultimate man-machine interface?– Intelligent
personal agent– Sharing
situations
command button
fingerprint recognizer
reflectors photo detector
microphone
speaker
Page 43
[email protected] 43
Optical Bi-directional Communication
server
cameraLED
terminal
•Download•voice
•power supply
•Upload•position•direction•motion•(ID)
Page 44
[email protected] 44
My Button V1: CoBIT
• Compact Battery-less Information Terminal
• Terminal: solar cell + reflector + earphone
Page 45
[email protected] 45
My Button V1
• Server (environment) side: LED + camera(s)
Page 46
[email protected] 46
青森産のリンゴです。一個200円です。これ買いますか?
ありがとうございます。
アラスカ産のシャケです。
これを買いますか?
となりにはサンマがあります。
Application to Markets
Page 47
[email protected] 47
Environment
The Image of the Target System
User
device
Intelligent Content
Sensors and actuators
Situationrecognition
Query/orders
information
request
Information retrievalInformation processing
Voicerecognition
Semantic query
User modeling
Photo/radio communication
vision
Multiagent technology
Application toEmergency rescue
Page 48
[email protected] 48
Tokyo
WirelessLAN
My-Button
i-lidar
camera
intelligent contentsemantic web
voicecommand/query
voiceinformation
IR
The next train will leave from platformNo 2 to your right.
CoBIT
Page 49
[email protected] 49
The evolution of the communication technologies in Star Trek
CoBITmy buttons
conventional
Page 50
ApplicationsApplications
Page 51
[email protected] 51
Mass Support
• Information support for the whole city
• Examples– Global Car Navigation
• Optimal route planning using the current positions and destinations of all cars in the area
• Coordinated signal control
– Theme Park total coordination
– On-line Government• Direct democracy with the help of discussion and
decision support system
Page 52
[email protected] 52
Special project for earthquake disaster mitigation in urban areas(Ministry of Education and Science)
• Ad hoc network for emergency communications
• Multiagent simulation for– evacuation
– rescue
Page 53
[email protected] 53
Real-time dynamic ad-hoc network for large-scale disasters (earthquake)– Effective in
congested areas
– Alternative to other communication infrastructures• First 10-30 min.
– Connection to other lines when possible
Page 54
[email protected] 54
Grounding the Simulation
KernelGIS
Agents
Viewers
ComponentSimulators
Portable devices Robots
Rescue team
Real-worldsensors
Rescue HQ
Page 55
[email protected] 55
Summary
• We have to connect the digital world to the real world→ GROUNDING
• Multiagent concept is the key to ubiquitous computing– Grounding
– The core architecture
– Ubiquitous agents for human support– Many MA applications