Interactive tele-presence in exhibitions through web-operated robots

Interactive Tele-Presence in Exhibitionsthrough Web-operated Robots

P. Trahanias1 W. Burgard2 D. Hahnel2 M. Moors3

D. Schulz3 H. Baltzakis1 A. Argyros1

1Institute of Computer ScienceFoundation for Research and Technology - Hellas (FORTH)

GR-71110 Heraklion, Greece{trahania,xmpalt,argyros}@ics.forth.gr

2Computer Science DepartmentUniversity of Freiburg, Freiburg, Germany

{burgard,haehnel}@informatik.uni-freiburg.de

3Department of Computer ScienceUniversity of Bonn, Bonn, Germany{moors,schulz}@cs.uni-bonn.de

Abstract

The current paper presents techniques that facili-tate mobile robots to be deployed as interactive agentsin populated environments, such as museum exhibi-tions or trade shows. The mobile robots can be tele-operated over the Internet and this way provide remoteaccess to distant users. Throughout this paper we de-scribe several key techniques that have been developedin the context of relevant EU-IST projects. The de-veloped robotic systems have been installed and exten-sively operated in the premises of various sites. Theuse of the above techniques, combined with appropri-ate authoring tools, has resulted in drastic reductionin the installation times. Such demonstrations ascer-tain the functionality and reliability of our methodsand provide evidence regarding the effectiveness of thecomplete systems.

1 Introduction

Mobile robotics technology and its application invarious sectors is currently an area of high interest.Research in this field promises advanced developmentsand novelties in many aspects. Applications of mobilerobotics technology in public spaces can be found in afield that we informally term “robots in exhibitions”.

In this context, robots offer alternative ways for inter-active tele-presence in exhibition spaces.

Two recent EU-IST funded projects, namelyTOURBOT1 and WebFAIR2 address this goal.TOURBOT pursued the development of an interac-tive tour-guide robot able to provide individual accessto museums’ exhibits over the Internet. The resultsof TOURBOT were demonstrated through the instal-lation and operation of the system in various organi-zations. WebFAIR is an ongoing project; it builds onTOURBOT results and attempts to extend relevantdevelopments to the more demanding environmentsof trade shows. Additionally, WebFAIR introducestele-conferencing between the remote user and on-siteattendants and employs multiple robots, enabling si-multaneous robot control by multiple users.

In this paper we present highlights of the techniquesdeveloped in the above mentioned projects. Theycover various aspects of robots that are deployed inpopulated environments. Among them is a feature-based technique for mapping large environments, amethod for tracking people with a moving mobilerobot, and an approach to filter spurious measure-ments coming from persons moving in the environmentwhile the robot is mapping it. Furthermore, we de-scribe new aspects of user/robot interaction and user

1http://www.ics.forth.gr/tourbot2http://www.ics.forth.gr/webfair

Proceedings of ICAR 2003 The 11th International Conference on Advanced Robotics Coimbra, Portugal, June 30 - July 3, 2003

1253

Figure 1: Line feature map (left) and occupancy grid map (right) of an exhibition site generated by the robot

interfaces. Among them are a speech interface for on-site users and a flexible web-interface with enhancedvisualization capabilities for remote users. Addition-ally, we report on the demonstration events that tookplace in the framework of TOURBOT and argue onthe drastic reduction of the system set-up time thatwas achieved.

2 Related Work

Over the last decade, a variety of service robotswere developed, designed to operate in populated en-vironments. Example cases are robots that are de-ployed in hospitals, museums, trade-fairs, office build-ings and department stores. In addition, a varietyof Web-based tele-operation interfaces for robots hasbeen developed over the last years. Three of the earliersystems are the “Mercury Project”, the “Telerobot onthe Web”, and the “Tele-Garden” [8, 9, 18]. Thesesystems allow people to perform simple tasks with arobot arm via the Web. Since the manipulators oper-ate in prepared workspaces without any unforeseen ob-stacles, all movement commands issued by a Web usercan be carried out in a deterministic manner. Addi-tionally, it suffices to provide still images from a cam-era mounted on the robot arm after a requested move-ment task has been completed. The mobile roboticplatforms Xavier, Rhino and Minerva [17, 4, 19] canalso be operated over the Web. Their interfaces relyon client-pull and server-push techniques to provide vi-sual feedback of the robot’s movements; this includesimages taken by the robot as well as a java-animatedmap indicating the robot’s current position. However,their interfaces do not include any techniques to mon-itor changes of the environment. 3D graphics visual-izations for Internet-based robot control have alreadybeen suggested by Hirukawa et al. [12]. Their inter-face allows Web users to carry out manipulation taskswith a mobile robot, by controlling a 3D graphics sim-ulation of the robot contained in the Web browser.

The TOURBOT and WebFAIR systems use video

streams to convey observed information to the user.Additionally, they provide online visualizations oftheir actions in a virtual 3D environment. This allowsthe users to choose arbitrary viewpoints and leads tosignificant reductions of the required bandwidth.

3 Feature-based Mapping

In order to navigate safely and reliably, a robotmust be able to find its position within its environ-ment. For this purpose, the creation and maintenanceof suitable representations of the environment is neces-sary. Two alternative mapping techniques have beendeveloped, that produce feature maps and occupancygrid maps, respectively.

The feature-based mapping algorithm utilizes linesegments and corner points which are extracted out oflaser range measurements [3]. During mapping, thepose of the robot is estimated via a hybrid localizationapproach, namely a switching-state-space model [2].At each (discrete) state, an Extended Kalman Filter(EKF) is used for accurate pose estimation.

To close loops during mapping, the algorithm inter-leaves localization and mapping just like other tech-niques which rely on the popular EM-algorithm [20].During the E-step, our algorithm uses the EKF toprovide a maximum a-posteriori estimate of the robotpose given all available measurements; in the M-stepthe mapped features are recalculated. The left imagein Figure 1 shows a typical map of an exhibition siteresulting from this process. During mapping the robotcould successfully close several cycles.

To perform several navigation tasks, such as pathplanning and obstacle avoidance, the TOURBOT andWebFAIR robots employ occupancy grid maps [13]and apply the probabilistic algorithms described in [4].The right image in Figure 1 shows a typical occu-pancy grid map that is learned from the same dataand used for the navigation while the robot providestour-guides.

1254

4 People Tracking

Tour-guide robots operate, by definition, in popu-lated environments. Knowledge of the position andthe velocities of moving people can be utilized in vari-ous ways to improve the behavior of tour-guide robots,e.g. by enabling a robot to adapt its velocity to thespeed of the people in the environment, by improv-ing its collision avoidance behavior and by facilitatinghuman-robot interaction.

The TOURBOT and WebFAIR systems applyprobabilistic data association filters (SJPDAFs) [15]to estimate the positions of people in the vicinity ofthe robot. A set of particle filters [14] is employed tokeep track of the individual persons. The particle fil-ters are updated according to the sensory input, usinga model of typical motions of persons. In the adoptedapproach, we compute a Bayesian estimate of the cor-respondence between features detected in the sensordata and the different objects to be tracked. The up-date phase uses this estimate to update the individualparticle filters with the observed features.

5 Mapping in Dynamic Environments

Learning maps has received considerable attentionover the last two decades. Although all approachespossess the ability to cope with a certain amount ofnoise in the sensor data, they assume that the environ-ment is almost static during the mapping process. Es-pecially in populated environments, additional noise isintroduced to the sensor data which increases the riskof localization errors or failures during data associa-tion. Additionally, people in the vicinity of the robotsmay appear as objects in the resulting maps and there-fore make the maps unusable for navigation tasks.Our mapping system is able to exploit the results ofthe people tracking process during the mapping pro-cedure [11]. This has several advantages. First, byincorporating the results of the people tracker, the lo-calization becomes more robust. Additionally, the re-sulting maps are more accurate, since measurementscorrupted by moving people are filtered out. Figure 2shows maps of the Byzantine and Christian Museumin Athens that were recorded with (left) and without(right) incorporating the results of the people-trackerinto the mapping process. Both maps were actuallygenerated using the same data set. While the robotwas acquiring the data, up to 20 people were moving inthis environment. The left image shows the endpointsof the laser-range data after localization. Obviously,a corresponding grid map would be useless, since it

would contain many spurious objects that might havea negative effect on several standard navigation taskssuch as localization and path planning. The right im-age of Figure 2 shows the map resulting from our ap-proach. As it can be seen from the figure, our tech-nique is able to eliminate almost all spurious objectsso that the resulting map provides a better represen-tation of the true state of the world.

6 The Web Interface

In addition to interacting with people in the ex-hibitions, a main goal in our projects is to establishtele-presence over the internet. Compared to inter-faces of other systems such as Xavier, Rhino and Min-erva [17, 5, 16], the web interface of the TOURBOTsystem provides enhanced functionality. Instead of im-age streams that are updated via server-push or client-pull technology, a commercial live streaming video andbroadcast software [21] is used. This software providescontinuous video transmission from the robot’s cam-eras to the remote user.

Additionally, web-users have a more flexible controlover the robot. They can control the robot exclusivelyfor a fixed amount of time (typically 10 minutes peruser). Whenever a user has control over the robot,he/she can direct it to either (a) arbitrary points inthe exhibition or (b) particular exhibits in the exhibi-tion. Furthermore, the user can select from a list ofpredefined guided tours. At each point in time, theuser can request a high-resolution image. This way,the interface combines the properties of previous sys-tems. In addition, the user can look towards arbitrarydirections by controling the pan-tilt unit of the robot.Last but not least, the user can request the robot toautonomously move around an exhibit in order to viewit from all possible directions.

7 Enhanced Visualizations

Since the robots operate also during opening hours,the robot has to react to the visitors in the museum.This makes it impossible to predict the robot’s courseof action beforehand. Therefore, it is highly impor-tant to visualize the environment of the robot and themoving people therein, so that the web user gets a bet-ter understanding of what is going on in the museumand why the robot is carrying out the current actions.

A typical way of providing information to theusers is video streams, recorded with static or robot-mounted cameras. This, however, has the disadvan-

1255

Figure 2: Maps of the Byzantine and Christian Museum in Athens created without (left) and with (right) peoplefiltering.

Figure 3: The enhanced 3D visualization allows arbitrary view-points. The left sequence shows the real and thevirtual view through the robot’s cameras. The right images show the robot guiding three people through themuseum and a bird’s eye view of the scene.

tage of limited perspectives and high bandwidth re-quirements. For these reasons, we developed a controlinterface, which additionally provides the user with avirtual reality visualization of the environment includ-ing the robot and the people in its vicinity. Based onthe state information received from the robot and ourtracking algorithm, our control interface continouslyupdates the visualization. Depending on the level ofdetail of the virtual reality models used, the Internetuser can obtain visualizations whose quality is compa-rable to video streams. For example, Figure 3 showstwo sequences of visualizations provided during theinstallation of the system in the Deutsches MuseumBonn in November 2001 along with images recordedwith a video camera and with the robot’s on-boardcamera. Within the graphics visualization, people areshown as avatars. As can be seen, the visualization isalmost photo-realistic and the animated avatars cap-ture the behavior of the people in the scene quite well.

8 System Demonstration

In the framework of the TOURBOT project a num-ber of demonstration trials was undertaken in the

premises of the participating museums. These demon-strations were combined with relevant events in orderto publicize and disseminate the results of the projectto professionals and the broader public. Factual infor-mation of these events is as follows:

• Foundation of the Hellenic World, Athens,Greece, May 28–June 2, 2001 (see Figure 4(a)).Exhibition: “Crossia, Chitones, Doulamades,Velades - 4000 Years of Hellenic Costume.” Theexhibition area comprised 2000 square meters.During the trial the robot operated approxi-mately 60 hours covering a distance of 14 kilo-meters. More than 1200 web users observed theexhibition through TOURBOT.

• Deutsches Museum Bonn, Bonn, Germany,November 6–11, 2001 (see Figure 4(b)). Ex-hibition: “Part of the permanent exhibition,highlighting scientific achievements that wereawarded the Nobel Prize.” The exhibition areain which the robot moved comprised about 200square meters. The system operated about 60hours, covering a distance of 10 km. Approxi-mately 1900 web visitors had a look around the

1256

(a) (b) (c) (d)

Figure 4: Snapshots of robots operating in various museums and fares. (a) Robot Lefkos operating in theexhibition of the Foundation of the Hellenic World. (b) Robot Rhino operating in the Deutsches Museum Bonn.(c) Robot Lefkos operating in the Byzantine and Christian Museum. (d) Robot Albert interacting with a personat the Heinz Nixdorf MuseumsForum (this picture is curtesy of Jan Braun, Heinz Nixdorf MuseumsForum).

museum via the robot.

• Byzantine and Christian Museum, Athens,Greece, December 3–7, 2001 (see Figure 4(c)).Exhibition: “Byzantium through the eyes of arobot.” The exhibition area comprised about330 square meters. During the trial the robotoperated 40 hours, covering a distance of 5.3kilometers. Given the large number of on sitevisitors, the TOURBOT team decided to devotemore tour-guide time to the on-site visitors asopposed to web visitors.

Additionally, TOURBOT was installed and oper-ated for a longer period of time (Oct. 2001–Feb. 2002)at the Heinz Nixdorf MuseumsForum (HNF) in Pader-born, Germany (see Figure 4(d)). This was in theframework of the special exhibition at HNF ”Com-puter.Gehirn” (Computer.Brain) with a focus on thecomparison of the capabilities of computers/robotsand human beings. Recently (June 2002), TOURBOTwas introduced for one week in the Museum of NaturalHistory of the University of Crete, Heraklion, Greece.

8.1 Installation Time

The large number of test installations ofthe TOUR-BOT system required sophisticated tools for the setupof the overall system. Obviously, the most crucial partis the generation of the navigation map. However,based on the techniques described above, the overallmapping process could in all cases be accomplishedwithin several hours. To avoid that the robot leavesits desired operational space or collides with obstaclesthat cannot be sensed, we manually create a secondmap with artificial obstacles. These artificial obstaclesare fed into the collision avoidance module [4] and thus

prevent the robot from moving into the correspondingareas.

A further time consuming process is the genera-tion of the multimedia-content that is presented tothe user for each exhibit. The TOURBOT system in-cludes a generic Multimedia database including html-pages, images, audio, and video sequences. Materialin the database can be changed and/or edited usingavailable software tools. Furthermore, the robot isequipped with a task specification that defines wherethe designated exhibits are and which content has tobe presented.

Most of the multimedia information pertinent tothe exhibits can be obtained directly from the exhi-bition sites, since pictures, text and other relevantmaterial are often already contained in existing Webpresentations. The whole setup can therefore be ac-complished in less than two days. This is an enor-mous speed-up compared to previous tour-guide sys-tems. More specifically, while Rhino and Minerva set-up times were 100 and 30 days respectively, the set-upof TOURBOT required only 1.5 days.

9 Conclusions

The goals set for by the TOURBOT and Web-FAIR projects are in-line with on-going activities to-wards the development of fully autonomous robotsthat operate in populated environments. The men-tioned projects aim at the development of interactivetour-guide robots, able to serve web- as well as on-site visitors. Technical developments in these projectshave resulted in robust and reliable systems that havebeen demonstrated and validated in real-world condi-tions. Equally important, the system set-up time hasbeen drastically reduced, facilitating its porting in new

1257

environments. Current research extends the naviga-tion capabilities of the robotic systems by addressingobstacle avoidance in the cases of objects that are notvisible by the laser scanner [1], 3D mapping [10], map-ping in dynamic environments [11], predictive naviga-tion [6], and multi-robot coordination [7].

Moreover, in the context of the above projects ad-ditional issues are addressed that consider (a) how toadapt this technology in order to fit the long-term op-erational needs of an exhibition site, (b) how to eval-uate the robotic system in terms of its impact to themain function and objectives of the exhibition site (fi-nancial impact, accessibility, marketing and promo-tion, impact on visitor demographic, etc.), and (c)how to evaluate the content and educational addedvalue to museum and exhibition visitors, and generatea feedback to the technology developers in order toimprove in the future the robotic avatars and adaptfurther to the needs of the users.

Acknowledgments

This work has partly been supported by the ISTProgramme of Commission of the European Com-munities under contract numbers IST-1999-12643 andIST-2000-29456. The authors would also like to thankthe members of the IST-project TOURBOT for help-ful comments and fruitful discussions.

References

[1] H. Baltzakis, Argyros A., and P. Trahanias, Fusion ofrange and visual data for the extraction of scene struc-ture information, Intl. Conf. on Pattern Recognition,(ICPR), 2002.

[2] H. Baltzakis and P. Trahanias, Hybrid mobile robotlocalization using switching state-space models, IEEEIntl. Conf. Robotics & Automation (ICRA), 2002.

[3] , An iterative approach for building featuremaps in cyclic environments, IEEE/RSJ Intl. Conf.on Intell. Robots and Systems (IROS), 2002.

[4] W. Burgard, A.B. Cremers, D. Fox, D. Hahnel,G. Lakemeyer, D. Schulz, W. Steiner, and S. Thrun,Experiences with an interactive museum tour-guiderobot, Artificial Intelligence 114 (1999), no. 1-2.

[5] W. Burgard and D. Schulz, Robust visualizationfor web-based control of mobile robots, Robots onthe Web: Physical Interaction through the Internet(K. Goldberg, R. Siegwart, eds.), MIT-Press, 2001.

[6] A. Foka and P. Trahanias, Predictive autonomousrobot navigation, IEEE/RSJ Intl. Conf. on Intell.Robots and Systems (IROS), 2002.

[7] D. Fox, W. Burgard, H. Kruppa, and S. Thrun, Aprobabilistic approach to collaborative multi-robot lo-calization, Autonomous Robots 8(3) (2000).

[8] K. Goldberg, S. Gentner, C. Sutter, J. Wiegley, andB. Farzin, The mercury project: A feasibility study foronline robots, Beyond Webcams: An Introduction toOnline Robots (K. Goldberg, R. Siegwart, eds.), MITPress, 2002.

[9] K. Goldberg, J. Santarromana, G. Bekey, S. Gentner,R. Morris, J. Wiegley, and E. Berger, The telegarden,Proc. of ACM SIGGRAPH, 1995.

[10] D. Hahnel, W. Burgard, and S. Thrun, Learning com-pact 3d models of indoor and outdoor environmentswith a mobile robot, 4th European workshop on ad-vanced mobile robots (EUROBOT’01), 2001.

[11] D. Hahnel, D. Schulz, and W. Burgard, Map build-ing with mobile robots in populated environments,IEEE/RSJ Intl. Conf. on Intell. Robots and Systems(IROS), 2002.

[12] H. Hirukawa, I. Hara, and T. Hori, Online robots, Be-yond Webcams: An Introduction to Online Robots(K. Goldberg, R. Siegwart, eds.), MIT Press, 2002.

[13] Hans P. Moravec and A.E. Elfes, High resolution mapsfrom wide angle sonar, IEEE Intl. Conf. on Robotics& Automation (ICRA), 1985, pp. 116–121.

[14] M.K. Pitt and N. Shephard, Filtering via simulation:auxiliary particle filters, Journal of the American Sta-tistical Association 94 (1999), no. 446.

[15] D. Schulz, W. Burgard, D. Fox, and A.B. Cremers,Tracking multiple moving objects with a mobile robot,IEEE Conf. on Computer Vision and Pattern Recog-nition (CVPR), 2001.

[16] D. Schulz, W. Burgard, D. Fox, S. Thrun, and A.B.Cremers, Web interfaces for mobile robots in publicplaces, IEEE-Magazine on Robotics & Autom. (2000).

[17] R. Simmons, R. Goodwin, K. Haigh, S. Koenig, andJ. O’Sullivan, A layered architecture for office deliv-ery robots, 1st Intl. Conf. on Autonomous Agents(Agents), 1997.

[18] K. Taylor and J. Trevelyan, A telerobot on the WorldWide Web, National Conf. of the Australian RobotAssociation, 1995.

[19] S. Thrun, M. Beetz, M. Bennewitz, W. Burgard, A.B.Cremers, F. Dellaert, D. Fox, D. Hahnel, C. Rosen-berg, N. Roy, J. Schulte, and D. Schulz, Probabilis-tic algorithms and the interactive museum tour-guiderobot Minerva, J. Robotics Res. 19 (2000), no. 11.

[20] S. Thrun, W. Burgard, and D. Fox, A probabilis-tic approach to concurrent mapping and localizationfor mobilerobots, Machine Learning and AutonomousRobots (joint issue) (1998), no. 31/5.

[21] http://www.webcam32.com/.

1258