Wireless wires: let the user build the ubiquitous computer

Wireless Wires: Let the User Build the Ubiquitous Computer

Davide CarboniCenter for Advanced Studies, Research and Development in

Sardinia

Parco Scientifico e Tecnologico, POLARIS,

Edificio 1, 09010 PULA (CA - Italy) +39 07092501

[email protected]

Pietro ZanariniCenter for Advanced Studies, Research and Development in

Sardinia

Parco Scientifico e Tecnologico, POLARIS,

Edificio 1, 09010 PULA (CA - Italy) +39 07092501

[email protected]

ABSTRACTIn this paper we describe the motivations, the rationale and a solution to build a web of things in which people have the important role to build new connections between real objects using a simple interaction based on pointing visual tags. Some scenarios are described and in particular the conference room scenario is analyzed in depth and a real prototype that implements this scenario is showed.

Categories and Subject DescriptorsC.2.4 [Distributed Systems]: Distributed applications.

General TermsDesign, Experimentation.

Keywordsubiquitous computing, mobile computing, proximity technologies, visual tags, distributed systems, web of things, internet of things.

1.INTRODUCTIONThe preeminent interaction paradigm for the masses is the desktop but this metaphor is no longer adequate to support both the complexity of data and the mobility of users [1]. On the other hand, artificial worlds like Second Life are the shared, online, social implementation of a virtual universe and, outside the value provided by user collaboration and the resulting virtuous network effect, the interaction flow is between a user body and a 2D screen on a desk. This draws too much the attention from the real world to the virtual one. Eminent authors do not consider virtual reality the right interaction paradigm for the pervasive computing claiming that it is better a embodied virtuality rather then a

virtual reality [2] or in other words: a web of real things rather than digital things in the web. In [3] sentient computers are able to reconstruct a virtual counterpart of the real environment without forcing the user to be immersed in a virtual reality. The environment is a model that machines can use to reconfigure themselves and to adapt at their best to the actual scenario of use.Invisible computing aims at making the computer as invisible as possible [4] letting it to blur in the background. Interacting with a computer is classified “low priority” task while higher priorities are reserved to interactions with people and with the problems in our application domain. Nobody of us would be happy to spend two hours to setup a word processor while our goal is writing an article. Making computers invisible is good but the problem with implicit interaction is that users loose control of the consequences of their interaction and they are not even aware that data is collected, processed, decisions are taken and system commands are launched. This frees the user from the burden of knowing low level activities but poses a real issue about privacy and monitoring of personal behavior.In Oxygen[5] a complete infrastructure is devised ranging from devices, networks technologies, perceptual technologies, automated tasks, collaboration and knowledge discovery. In Tangible bits [6] the idea is to make the user aware of surrounding computing capabilities exposing the interaction to real graspable objects. The tangible user interfaces employ physical objects, surfaces, and spaces as tangible embodiments of digital information.Paraphrasing Albert Einstein, we claim that the user interaction “must be as simple as possible but not simpler”. Users should have the power with comes with responsibility to draw connections between everyday objects. There is control and the interaction is explicit. The main contribution of this paper is the description of a system that allows users to build networks of everyday objects using proximity technologies.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.MUM’07, December 12-14, 2007, Oulu, Finland.Copyright 2007 ACM 978-1-59593-916-6/07/0012…$5.00.

2.PROGRAMMING THE UBIQUITOUS COMPUTERRationaleThe web is, among the others, a proof that a simple mechanism applied to a large number of individuals allows a more complex system to emerge [7][8]. So that the web today is much more than billions of pages linked together but it is one of the greatest achievement of the human history.Simple pages linked each other have transformed the act of reading in a sort of interactive play where users can access instantaneously every content presented according to his actual and personal needs and preferences. The web is basically built on two metaphors: the hypertext and the hyperlink. The former is the just a digital reification of human language in its written form, while the second is a mechanism that non-digital forms of writing (like writing on paper) could not provide. The two metaphors have fostered the huge development of the web in a form of web of pages and both hypertext and hyperlink belong to the digital world. In a service oriented architecture not only web pages are linked together but also information services. Yahoo! Pipes [9] is an example of how digital objects (like RSS feeds) can be linked together to form new aggregation of services in which the pipes are user defined. We want to extend this piping paradigm not only to digital objects but also to real ones and build a web of things. Quoting Metcalfe [10 ] :

“A ‘World Wide Web of Things’ would enable easy publishing and spontaneous viewing of control-sense-identify information. A ‘Google of Things’ would interconnect CSI data to enable powerful search and inference.”

The today follow-up of the great world wide web revolution is the web of participation, also called the web of people, or social network's web. In simple but effective way it was christened web 2.0 by Tim O'Reilly[11]. A discussion about the web 2.0 is outside the scope of this paper and the only point that we want to highlight here is that in web 2.0 the emergence of a new complex system is based on the participation and active behavior of millions of users.The effort to assign the web 3.0 brand is now ongoing [12][13] and many are proposing the semantic web[14] as the successful candidate[15]. The scientific literature about semantic web is luxuriant and the academic effort is noticeable but users are not probably perceiving the importance of such undergoing work. On the other hand web 2.0 has been immediately visible for the average user and it can coexist with the semantic web [16]It is a matter of fact that the semantic web, that the its inventor today prefers to rename the data web [17], is the next big step ahead. In our opinion, the whole picture in which machines communicate among them exchanging structured data the role of users needs to be better defined. Like in web 2.0 the novelty is not in technologies but rather in the way such technologies are exploited, even in ways not foreseen by the inventors of such technologies. Aside the web of pages and the web of user communities, another web can coexists: the web of things. This vision drives the work described in this paper and is not in

contrast with semantic web but rather is a natural application of its technologies where users and everyday things are the building blocks of the interaction flow.

Figure 1: Possible interconnections: web pages, tags, user identities, web services and “things” can be interconnected each other.

We assume a scenario in which every object is connected to a pervasive wireless/wired network and can answer to a HTTP query with structured data. Everyday surrounding objects like phones, domestic appliances, advertisement billboards, musical instruments become the nodes of the web of things. It is not up to engineers to decide whether or not a guitar must be connected to a refrigerator. This is up to users. Under these technological assumptions, not so far from the today reality, users, objects and network are the ingredients to build a web of things in which users are also the programmers (Figure 1).

Table 1. Chronological stages of the Web. Every stage is partially based upon the previous ones and emerges like a new level of complexity

web 1.0 made of pages linked

web 2.0 made also of users, user generated content, and tags in social networks

web 3.0Services connect and exchange structured data. The semantic/data web is the right

playground.

web of things

made also of real (as opposite to digital) objects able to interact among them and with users. The invisible wires between objects are

built by users and not predetermined in the design.

The Importance of ProximityThe point-click-and-download paradigm has already been ported to the physical world: two-dimensional barcodes systems like Shotcode, Semacode, and QR can be attached with no cost to any object and appropriate browsers can recognize codes and download linked information from the Internet. In this paper we claim that the use of visual or radio tags to simply download a media file to the user phone is a too simplistic way to exploit

these technologies and we want to show how to enable a new class of applications. The proximity of users to objects is a fundamental aspect that must be considered in computing. In [18] the authors use Bluetooth as option for providing connectivity, and propose RFID technology to enhance the Bluetooth connection establishment procedure. In [19] the authors experiment the selection of a website by physically pointing at its symbol and making a phone call by pointing at an icon representing the person to be called. Their system based on smart phones and infrared sensors was perceived more positively by the users than a conventional method.Our approach to proximity is that after an object and in a given situation and with a given mood a person can have the idea to build something new. The subsequent action in our scenario is to build a connection. If we imagine the world as a giant sketch board we just want a way to draw a line from an object to another and build something useful as result. Simple but effective rules, applied to a multitude of objects tend to form a complex system. In our scenario millions of real objects can simply be connected with natural gestures in the real environment without sitting in front of a PC screen. We can call hyperpipes (Figure 2) these links. We use the prefix hyper to denote that they are virtual as opposite to the reality of the objects they are linking. Through hyperpipes objects exchange data using appropriate protocols.

Terminologyobject: any object either real or digital addressable with a HTTP query that can exchange data and can be networked.

sink: a part of an object able to receive, process and make use of data received. For instance, an object like a TV box can have a sink able to receive a video stream.

source: a part of an object able to transmit data upon request either in pull or push modality depending on the underlying transmission protocol. For instance, an RSS feed can be a source with pull modality.

hyperpipe: (or simply pipe) a logical connection between two objects. A pipe is completely described by its state and by two endpoints. The two endpoints of a pipe are always a source that transmits and a sink that receives.

identification: it is the process in which an object is identified by means of a proximity technology. Visual tags or radio tags can be used fo this purpose. After the identification follows the selection of sink/source available for that object.

registration: it is the process in which an object advertises in a registrar its presence and its sinks/sources. The registration could be avoided if the mobile sensor can receive from the identification procedure all data needed to establish a new hyperpipe.

mobile sensor: it is the sensor used to point an object, select sinks/sources and establish an hyperpipe. In our scenarios this role is always performed by a camera phone.

Connecting objects in the physical environment with a gestureEach object must be identified by the user before it can be connected in a hyperpipe. Once the identification is done the user can select either a source or a sink as endpoint and grab the endpoint to another object to build an hyperpipe. Identification technologies can consist in radio or visual tags recognizer installed in users' camera phones. Camera phones are probably the right choice because are of common use and can provide a minimal but useful interactivity to setup hyperpipes. When the user identifies an object with the recognizer software, an interactive setup asks the users if he wants to initialize a new hyperpipe. Latter, when the user identifies a second object the setup interactive software will ask if the previously initiated pipe must be completed including the second endpoint.It is mandatory that the two objects involved in a hyperpipe have one a data source and the other a data sink. In general an object can have N different sinks and M different sources and can be part of different hyperpipes at the same time. When an object has more than one sink/source the mobile sensor must present a selection task to the user in order to select among multiple possibilities.In particular, we consider useful for our scenarios objects with no general purpose computing capabilities, but appliances with a precise function like billboards, refrigerators or boilers. More in general, a general purpose PC is an object with many sources like screen, keyboard and every running process in it could be viewed both as a data source (i.e. the stdout or the GUI can be transmitted through the network) and like a sink (its standard input).

3. SCENARIOS OF USEIn this section some scenarios of use are sketched. They are based on the assumption mentioned in the section 2: objects are all connected to the ubiquitous network.

AnemometerAn anemometer is an instrument to measure the wind speed and direction. In our scenario it is addressable with HTTP queries and we can know the speed and the direction in some XML format.

http://someaddress /getwindspeed?format=rss

The choice of XML and RSS is just to make the example more realistic. This address is a data source of the anemometer. Then we consider a second object: a printer at home. Transforming the printer in a RSS reader is simple, but it is interesting to register the feed in the printer using natural interaction instead of some control panel in our PC. The novel approach is to use our camera phone to point the anemometer, initialize an hyperpipe, and then at home complete the connection pointing our printer. A possible dialog could be:(...when the user points the anemometer)

phone: This is a data source. Do you want to initiate a pipe?

user: YES

phone: Do you want to set a filter?

user: YES

phone: minimum speed (knots)

user: 15

phone: OK. Pipe initiated. Seek a sink please...

( ...sooner or later depending on the user)

phone: This is a printer. It has a data sink. Do you want to complete the pipe?

user: YES

phone: Ok. Pipe complete. Printer is connected to anemometer. From now on every time the wind blows beyond 15 knots a sheet of paper is printed with this news. The interesting point to notice here is that in the dialog the user sets a filter on the wind speed (speed >= 15 knots). This filtering makes sense in this hyperpipe but does not in general (it is useless if you are connecting a refrigerator to your alarm clock). Then the dialog is composed by a fixed part not depending from the objects connected and by a specialized part that dynamically is loaded and presented in the camera phone to the user. The specialized fragment of dialog can be loaded from the network or, if the tagging technology is capable enough, directly encoded in the visual or radio tag.

Figure 2: State diagram for hyperpipes. When a first endpoint is selected by the user a new hyperpipe is initiated. With a second endpoint the hyperpipe is complete and ready to be flowed by data.

The Conference RoomIn this scenario the user is waiting to speak to a public event where a large digital board is the screen for the audience. The user is still working on the last slide and when it is his turn to talk. He simply uses the phone to point at his PC. A possible dialog will follow:(...the user points the laptop screen)

phone: This is a laptop screen XYZ. Do you want to select it as data source?

user: YES

phone: OK. Pipe initiated. Seek a sink please...

(...then the user points at the large screen in the conference room)

phone: This is a conference screen ZYX. It has a data sink. Do you want to complete the pipe?

user: YES

phone: Ok. Pipe complete. Conference screen is connected to laptop screen XYZ.

From that moment on the conference screen receives what is displayed in the laptop screen. The protocols can be RTP, VNC or whatever suitable to transmit a PC screen as a stream.

Figure 3:Conference room prototype: the user builds a connection between the laptop screen and the board. From now on, the laptop screen is transmitted to the board.

To complete his job our user prefers to use a pointing device that better suits for a talk. For instance a device with accelerometers like a Nintendo Wiimote(TM). So he points at his Wiimote and starts the dialog

(...the user points the wii)

phone: This is a wii. Do you want to select it as data source?

user: YES

phone: OK. Pipe initiated. Seek a sink please...

(....the user points at the mouse)

phone: This is a mouse device YKX. It has a data sink. Do you want to complete the pipe?

user: YES

phone: Ok. Pipe complete. Wii is now connected to mouse YKX.

Considerations about the conference room scenarioA secondary assumptions here is that also parts of a complex system like a PC are independent objects that can be networked with hyperpipes. In this scenario the PC screen is used as data source and the mouse as data sink. What is produced by the screen is transmitted to the conference screen and what is produced by the Wiimote is transmitted and translated to mouse events. The conference room scenario is similar to a Bluetooth use case in which a device (the PC) is connected to some other peripherals. Nevertheless, the possibilities of our architecture are quite different:

objects can be networked even if they are not peripherals

objects are not necessarily connected through a wireless link

objects are discovered and connected because users draw a connection in a drag and drop fashion.

Another point worth noticing is that not only real objects can be recognized and used in pipes but also digital one like web pages or RSS feeds. In fact, the pointing mechanism could be augmented to recognize the web address in our browser and then use it like an endpoint in a pipe. Plug-ins that produce QR tags on-the-fly from a web page are already available in Firefox [20]

Other scenariosHere we list just some applications:

1. mobile phone –> phone: A call redirect from mobile to wired phone. It is useful if the user does not want to use his mobile phone for incoming calls.

2. Phone -> mobile phone: The opposite of the previous one. When the user leaves his desk all calls will be redirected to his mobile.

3. Podcast link in the PC screen -> TV: A podcast registration is made from a web page to a media center in the living room.

4. boiler -> TV: (with a filter), so when the water is warm enough a subtitle in the TV screen says it is shower time.

5. mobile phone (with accelerometer) -> engine equipped videocamera -> mobile phone screen: In this case the movement of the phone controls the engine of the camera and the video stream is transmitted back to the mobile screen.

6. mobile phone (with accelerometer) -> engine equipped videocamera -> mobile phone screen> TV screen: like the previous one but the video is also transmitted to a large screen and shared with people in the same room.

4.THE PROTOTYPEOur objective was to build a prototype for the conference room scenario. Figure 3 and 4 shows our prototype at work. The architecture of the prototype is depicted in Figure 5.

Figure 5: Minimalist architecture of hyperpipe system: the mobile sensor (cameraphone) is used to identify an object. As return it receives from the manager the Ids of one or more sinks/sources. The mobile sensor can init a new pipe or complete a newly created pipe. Objects must register in the manager at boot and check regularly for newly created pipes.

Object identification

The first important step is to integrate in the prototype a technology to identify objects. We chose to use visual tags, and in particular, the QR standard [21].The QR tags can encode a quantity of data depending on the size of the tag. With a stamp size tag is possible to encode about one hundred bytes. Visual tags have two immediate advantages:

producing a new tag is a no-cost operation

Figure 4: Conference room prototype: the user builds a connection between the laptop mouse and the Wiimote. From now on, the movement of the Wiimote fires mouse events to the GUI.

any camera phone can be in principle be programmed to recognize tags.

On the other hand, radio tag identification is available only with few phone models. With respect to other barcode encodings, QR have some advantages:

it is an open standard: The Japanese standard for QR Codes, JIS X 0510, was released in January of 1999, and a corresponding ISO International Standard, ISO/IEC 18004, was approved in June of 2000.

QR codes use the Reed–Solomon error correction and it is quite robust to image distortion. It is also possible to embed in the code some images or logos without loosing any information of the code.

It is very popular and it is quite easy to find encoders. A lot of mobile decoders are also freely available but they are not open sourced.

Pipe ManagerIt is a component of our architecture which has two main roles:

it stores a list of available objects in the network. So it acts as a device/sink/source registrar.

It also stores a list of active or activating hyperpipes.These functionalities are available through REST web services so that objects can register their sinks/sources in the service registrar and the mobile sensor can connect to initialize and complete hyperpipes.Notice that the architecture is centralized around the manager and this is a vulnerability that we are aware of. But, our prototype is at the moment only useful to show the interactions between people and objects in a limited environment. To make the system able to scale globally the manager architecture should be revisited.For instance, objects endpoints data could be directly encoded in visual tags without the need to use a central service registrar. On the other hand, an object with a lot of sinks and sources cannot easily encode this information in a small visual tag. Other proximity technologies could be used instead like Near Field Communication (NFC). Also the pipe status is maintained in the manager, this could be avoided in principle using some decentralization technique resulting from the P2P and distributed computing research efforts. Thus, to test our idea we designed and implemented a minimalist service infrastructure. When a visual tag is captured and recognized by the mobile sensor the user selects which sink or source to be used in the pipe. When the user selects a second object the manager proposes one among the possible sinks or sources available for that pipe. For instance, if the first endpoint is a data source with VNC protocol then for the second object the manager asks the user to select among sinks compatible with VNC. The order of the selection is not relevant: the user can initiate a new pipe either selecting a sink or a source. Online services able to adapt a protocol to another could be used to pipe together sink and sources that are in principle not able to communicate. Then when a pipe is completed with two endpoints, it is registered in the pipe manager. Objects must regularly check against the manager if they are involved in a new active pipe and in that case they must start the appropriate procedure and protocol handshake to establish a peer to peer data transmission.

For instance, if the source A is a RSS feed and B is a sink able to process RSS the mechanism is a simple pull of data: first O checks with the manager if a new pipe is active with one of its sinks/sources. It verifies that its sink B is now involved in a new pipe and O registers in its local memory A as new RSS feed and starts pulling every now and then the XML to be processed.

The object O queries the manager to know in which hyperpipes its sink B is involved:

http://server/pipe?cmd=query&sink=B

and gets as result: <pipes><pipe pipeid="AB" sink="B" source="A" /></pipes>then the object O invokes the query: http://server/endpoint?id=A and gets:<endpoints><endpoint description="Laptop" deviceID="4" id="A" ip="x.y.z.k" protocol="vnc" type="output"/></endpoints>

The above code fragment could be the query and the response of the manager when an object wants to open the pipe for data to flow.A technical challenge is how to provide a flexible way to connect sinks and sources in a way that a shared protocol is chosen and that the most suitable version of that protocol is used accordingly to the capabilities of both sink and source. We are not facing this issue with novel techniques in our prototype because we do not want to reinvent the wheel: we are confident that existing frameworks like UPnP [22] or equivalents can be used for this purpose.

5.CONCLUSIONIn this paper we describe the motivations, the rationale and a possible solution to build a web of things in which users have the important role to build new connections between objects acting both as users and programmers. Our system is based on the assumption that users must be enabled with proximity technologies to build connections that we call hyperpipes. Proximity technologies like visual tags are inexpensive and can be easily experimented with off-the-shelf camera phones. Some scenario are described and in particular the conference room scenario is analyzed in depth and a real prototype that implements this scenario is showed.A first release of the prototype is available but in this early stage it has been tested with technicians only. The next step is to deploy a more user friendly point and setup interaction in the camera phone to perform a complete user trial.

6.REFERENCES[1] Mark Weiser. "The world is not a desktop". Interactions;

January 1994; pp. 7-8. [2] Mark Weiser, "The Computer for the Twenty-First Century,"

Scientific American, pp. 94-10, September 1991 [3] Addlesee, M., Curwen, R., Hodges, S., Newman, J.,

Steggles, P., Ward, A., and Hopper, A. 2001. Implementing a Sentient Computing System. Computer 34, 8 (Aug. 2001), 50-56. DOI= http://dx.doi.org/10.1109/2.940013

[4] Gaetano Borriello, "The Challenges to Invisible Computing," Computer, vol.33, no.11, pp. 123-125, Nov., 2000

[5] Oxygen - http://www.oxygen.lcs.mit.edu/Overview.html[6] Ishii, H. and Ullmer, B. 1997. Tangible bits: towards

seamless interfaces between people, bits and atoms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, United States, March 22 - 27, 1997). S. Pemberton, Ed. CHI '97. ACM Press, New York, NY, 234-241. DOI= http://doi.acm.org/10.1145/258549.258715

[7] J.H. Holland, Emergence: from chaos to order, Helix Books, New York (1998).

[8] http://en.wikipedia.org/wiki/Metcalfe%27s_law[9] Yahoo! pipes. http://pipes.yahoo.com/pipes/docs[10] A Google of things. http://www.automationworld.com/view-

2413[11] Tim O'Reilly. What is Web 2.0. Tim O'Reilly personal blog.

http://www.oreilly.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html

[12] Web 3.0. http://en.wikipedia.org/wiki/Web_3[13] Web 3.0 = Semantic Web? http://techbiz.blog.com/1730241/

[14] The Semantic Web- Tim Berners-Lee, James Hendler, Ora Lassila. Appeared in: Scientific American 284(5):34-43 (May 2001)

[15] Lassila, O. and Hendler, J. 2007. Embracing "Web 3.0". IEEE Internet Computing 11, 3 (May. 2007), 90-93. DOI= http://dx.doi.org/10.1109/MIC.2007.52

[16] Ankolekar, A., Krötzsch, M., Tran, T., and Vrandecic, D. 2007. The two cultures: mashing up web 2.0 and the semantic web. In Proceedings of the 16th international Conference on World Wide Web (Banff, Alberta, Canada, May 08 - 12, 2007). WWW '07. ACM Press, New York, NY, 825-834. DOI= http://doi.acm.org/10.1145/1242572.1242684

[17] Interview with Tim Berners-Lee , Business Week, April 2007. [18] Salminen, T., Hosio, S., and Riekki, J. 2006. Enhancing

Bluetooth Connectivity with RFID. In Proceedings of the Fourth Annual IEEE international Conference on Pervasive Computing and Communications (Percom'06) - Volume 00 (March 13 - 17, 2006). PERCOM. IEEE Computer Society, Washington, DC, 36-41. DOI= http://dx.doi.org/10.1109/PERCOM.2006.23

[19] Ailisto, H., Pohjanheimo, L., Välkkynen, P., Strömmer, E., Tuomisto, T., and Korhonen, I. 2006. Bridging the physical and virtual worlds by local connectivity-based physical selection. Personal Ubiquitous Comput. 10, 6 (Sep. 2006), 333-344. DOI= http://dx.doi.org/10.1007/s00779-005-0057-0

[20] Firefox Mobile Barcoder. https://addons.mozilla.org/it/firefox/addon/2780

[21] QR Code.com. http://www.denso-wave.com/qrcode/[22] The UPnP Forum: http://www.upnp.org/