Scalable Maps of Random Dots for Middle-scale …visgraph.cse.ust.hk/projects/random-dots-map/vr2013rdmm.pdfScalable Maps of Random Dots for Middle-scale Locative Mobile Games ...

To appear in IEEE Virtual Reality 2013 proceedings

Scalable Maps of Random Dots for Middle-scale Locative Mobile GamesLu Chen1∗ Hongbo Fu2† Wing Ho Andy Li2‡ Chiew-Lan Tai1§

1Hong Kong University of Science and Technology 2City University of Hong Kong

ABSTRACT

In this work we present a new scalable map for middle-scale loca-tive games. Our map is built upon the recent development of fidu-cial markers, specifically, the random dot markers. We propose asimple solution, i.e., using a grid of compound markers, to addressthe scalability problem. Our highly scalable approach is able togenerate a middle-scale map on which multiple players can standand position themselves via mobile cameras in real time. We showhow a classic computer game can be effectively adapted to ourmiddle-scale gaming platform.

Index Terms: H.5.1 [Information Interfaces And Presentation]:Multimedia Information Systems—Artificial, augmented, and vir-tual realities; I.4.9 [Computing Methodologies]: Image Processingand Computer Vision—Applications

1 INTRODUCTION

Middle-scale locative games are games where the gameplay relieson the players’ physical location in a classroom-size space (Fig-ure 1). In contrast to small-scale Augmented Reality (AR) games(e.g., [6, 11, 16, 17]) or large-scale locative games typically basedon Global Positioning System (GPS) (e.g., [13, 3, 2]), middle-scalelocative games require physical movement in an environment oftalking distance, thus providing unique gaming experience.

Middle-scale locative games pose a challenge in localization ac-curacy. GPS or many other positioning techniques (e.g., based oninfrared, WLAN/Wi-Fi or ultra-wide-band) are not suitable for suchgames, due to their limited accuracy, which has been reported atbest in the range of 3-6 meters for indoor usage [9]. In contrast,computer vision-based localization techniques could afford veryhigh accuracy but have been mainly applied to small-scale envi-ronment due to their high sensitivity to feature occlusion. Thereexist little research on middle-scale locative gaming platforms [5].

A feasible solution to the localization problem for middle-scalelocative games is to assemble a map of AR markers. Our work isbased on the random dot markers recently proposed by Uchiyamaet al. [14], which are much less visually obtrusive and more ro-bust to occlusion than classic squared-shaped fiducial markers (e.g.,ARTag [4]), as adopted in [5]. However, since each random dotmarker requires certain amount of memory for storing the LLAH(Locally Likely Arrangement Hashing) descriptors [10], naı̈velyforming a grid of distinct random dot markers or using a singlerandom dot marker with sufficient dots does not scale well to themiddle-scale platform due to the limited memory on mobile de-vices. In order to achieve high scalability while maintaining afford-able memory consumption, our map is constructed by reusing therandom dot markers in a smart way.

We build a prototype of middle-scale locative game platformbased on scalable maps of random dots serving as the game map

∗e-mail: [email protected]†e-mail: [email protected]‡e-mail: [email protected]§e-mail: [email protected]

Figure 1: Illustration of middle-scale locative games based on scal-able maps of random dots.

(Figure 1), where multiple players can stand and position them-selves via mobile cameras in real time (see the accompanyingvideo). To avoid feature occlusion by players, individual playersface their cameras downwards and capture the map region aroundtheir own feet, forming their own “private” but dynamic regionsfor feature matching and tracking. Our system does not requireany special hardware sensors and supports both indoor and outdoormiddle-scale locative games. It is fast and its accuracy is control-lable by varying key parameters of the dot marker configuration.

2 RELATED WORK

Barba and MacIntyre [1] present a scale model concept, wheremixed reality experience is categorized into figural, vista, environ-mental and geographical spaces based on spatial scale and its asso-ciated cognitive processes. By their criteria, our middle-scale gam-ing experience bears some resemblance to a vista space experience,which is spatially larger than an adult human body but could be vi-sually apprehended from a single place without moving. However,the merits of our platform reside in the locomotion of multi-userswithin physical maps, enabling locative gaming experience, whichis a typical feature of environmental space applications. This makesour system fall between the vista and environmental spaces by theirdefinitions, demonstrating a more delicate understanding of spatialscale of mixed reality experiences.

There exist many small-scale vision-based AR games, e.g., theInvisible Train by Wagner et al. [16], ARTennis by Henrysson etal. [6], BragFish by Xu et al. [17] and XNA Racing Game by Oda etal [11]. They typically require one or multiple players to surrounda grid of markers and point their cameras to the markers. Sucha setup allows every player to easily access the markers but over-constrains the space for physical movement, typically resulting incircular movement of the players around the markers.

Using features in a natural scene for localization seems a promis-ing direction for middle-scale locative games. However, such an ap-proach is highly sensitive to the availability of a rich set of featuresfor camera calibration. Thus it often does not work well in barelydecorated or textureless rooms (e.g., a classroom). Moreover, it isexpected that its performance or localization accuracy would varywith locations or views. Worse, rich features could be easily oc-cluded by players. Thus, existing works adopting this approach [8]often demonstrate only single-player scenarios.

Hence a more robust solution is to fully cover the environment

1

To appear in IEEE Virtual Reality 2013 proceedings

with maps of AR markers [5]. Ideally, such maps should have thefollowing properties: being as less obtrusive as possible, robustagainst occlusion (e.g., by the players’ own feet), highly scalable,and has controllable accuracy.

Unfortunately, none of the existing AR markers satisfies all theseproperties. Traditional square-shaped fiducial markers (e.g., thoseused in ARToolKit [7]) offer controllable accuracy and high scala-bility (e.g., as demonstrated in ARTag [4]). However, they are notrobust against occlusion and are visually obtrusive to be placed onthe floor as part of an entertaining environment. In contrast, prede-fined fully textured images as markers [18] are less obtrusive. How-ever, simply enlarging a single image to cover the interacting areacannot guarantee sufficient accuracy for localization, since only asmall portion of the entire image (i.e., private regions belonging toindividual players) would be viewed by individual cameras in oursetup. Forming a grid of images is a possible solution to this accu-racy problem but might be computationally too expensive.

The random dot markers proposed by Uchiyama et al. [14] tosome extent strike a balance in terms of the above desirable prop-erties. The random dot markers provide flexible marker design androbustness against occlusion (e.g., by the players’ own feet), mak-ing it ideal for our target platform. Another advantage of the ran-dom dot markers is that each dot is essentially a feature point, andthus by controlling properties like dot size and inter-dot distance, itis easy to control the feature density and accuracy of localization.

3 METHODOLOGY

A straightforward solution to using random dot markers for middle-scale platforms is simply to generate sufficient dots on a singlemarker, as similarly done in [12] for handwriting reconstruction,where a camera attached to the pen views the random dots scat-tered on a paper. However, our preliminary experiments reveal thatthis approach does not scale well in terms of memory usage. Forexample, to cover a typical middle-scale space of size 10m× 6m,i.e., 60m2, a single marker would need 12,000 dots and cost about210MB. Here we assume that about 200 dots per square meter needcaptured by the camera view at any place on the map for localiza-tion. In contrast, our approach needs only a quarter of that amountof memory. Noticeably, the latest Android phones typically al-low only a maximum heap size of 64MB, which prevents usingthe single-marker approach. As we will show below, the numberof markers needed in our approach only increases linearly in thesize of the map, i.e., O(w+ h), where w and h are the width andheight of the map respectively, while the straightforward approachis largely equivalently to an approach using O(w∗h) different mark-ers. Therefore, our solution is fast and highly scalable.3.1 Scalable Map Representation of Markers

We present a simple but effective solution to generate a middle-scale map of markers. The key idea is the use of compound markers,where two distinct basic markers, one with an odd marker ID andanother with an even marker ID, are laid on above another (eitherthe odd or the even marker on top). Every pair of odd and evenmarker IDs uniquely defines a compound marker and its location inthe map, with the odd IDs encoding the x coordinates and the evenIDs encoding the y coordinates.

We fill the odd rows of the map with markers with odd IDs andfill the even rows with markers with even IDs. To ensure that eachcompound marker constitutes different odd and even ID pairs, weinsert a row of markers all with identical even ID between everytwo rows of odd IDs markers (Figure 2). We use only two lists ofodd ID markers and re-use them at alternate rows. In this way, eachcompound marker is guaranteed to have a unique pair of odd andeven IDs. The number of distinct basic markers needed to generatea map with m rows and n columns is thus O(m+ n), which dra-matically decreases the memory consumption, making our methodsuitable for most mobile devices.

Figure 2: A map of compound markers, each consisting of two ran-dom dot markers, one with odd ID and one with even ID, representedas (odd ID, even ID) pair. (Best viewed in color)

The above-described map representation can be used togetherwith different kinds of markers. We adopt the random dot markerproposed by Uchiyama et al. [14] for its flexible marker design(supporting any shape) and, more importantly, its robustness againstocclusion. Each basic random dot marker is of a rectangle shape sothat the compound marker formed by two dot markers is a square.By varying three key parameters, namely the number of dots, dotradius and minimum inter-dot distance, the performance and accu-racy of localization can be controlled.3.2 Modified Marker Retrieval Algorithm

This section explains a variant of the original random dot markeralgorithm [14] to enable the detection and tracking of compoundmarkers.

To initialize the system, a hash table is built for fast retrievalof dot ID and marker ID. For each dot in the basic random dotmarkers in use, we calculate its LLAH descriptor based on localneighbor dots. Then the descriptor is hashed into a 1D index, whichis inserted into the hash table along with the dot ID and marker ID.

At runtime, for each input frame from the camera, we first bina-rize the frame, extract the centroid of each connected region as thedot center, calculate the LLAH descriptor for each dot, and retrievethe dot’s ID and its corresponding marker ID from the hash table.In order to accelerate this process on mobile devices, we resize eachframe into a quarter of its original size before binarization.

Next, we introduce a modified algorithm to identify all the in-dividual random dot markers from the retrieved dots. Due to ourrepeated usage of markers, the original algorithm needs to be modi-fied to detect individual basic markers for forming compound mark-ers for localization. We begin by clustering the dots according totheir marker IDs. At this stage, dots from markers with identicalIDs form a cluster (Figure 3). To separate these dots into individualbasic markers, we use an iterative method adapted from the orig-inal marker retrieval algorithm. In each iteration, we estimate theRANSAC-based homography between all the dots in the currentcluster and the stored dots set of the marker with corresponding ID.The resulting homography maps to one dot set in the current cluster.We then re-project the stored dots onto the frame and detect moredot correspondences. All the matched dots (dot ID and marker IDidentified) are then excluded from the next iteration. The iterationends when there are insufficient unmatched dots to form a marker(mostly peripheral dots). In this way, each basic random dot markerin the frame is associated with its set of dots. The location of eachbasic marker is the center of all its belonging dots.

Based on the locations of individual basic random dot markers,compound markers are detected to locate the camera in the map.

2

To appear in IEEE Virtual Reality 2013 proceedings

Figure 3: Illustration of the marker retrieval process. For each frame, we extract dot centers and retrieve each dot’s ID and its marker ID usingLLAH descriptors. After clustering the dots based on their marker IDs, individual basic markers are identified by iteratively estimating RANSAC-based homography until no more markers are detected. Finally, the compound markers are formed, from which the 3D camera pose and itsposition are estimated.

The detected basic markers are split into two lists, one with markerswith odd IDs and another with even IDs. Then, compound markersare formed by pairing the nearest two markers, one from each list.For each detected compound marker, its odd and even ID pair de-fines its location in the map. From this location, the corresponding3D dots positions in the real world are obtained. Using the camerapose estimation method of Uematsu et al. [15], 3D camera poseis estimated for each compound marker. For robustness against in-valid detection of compound markers, we compare the camera poseof each compound marker in the current frame and choose the onethat least deviates from the previous frame if available. By utilizingthe final camera pose, 3D camera location with respect to the realmarker map is easily obtained.

4 EVALUATION

4.1 Localization accuracyAs discussed before, our map of compound random dot mark-

ers uses significantly less memory than the single-marker solution.Below we compare their localization accuracy using synthetic im-age sequences. We first generated virtual camera walk-throughsof each map with various marker configurations using Maya. Ourconfigurations involve three key parameters: the number of dots permarker, the radius of dots and the minimum distance between thedot centers. The first parameter solely controls the dot density andthe other two parameters decide how evenly the dots are distributedin a marker. In our experiment, we varied the number of dots permarker and set the other two parameters optimally to evenly dis-tribute dots in the markers. Specifically, we fixed the dot radiousto 4 pixels and set minimum distance between the dot centers to45, 40, 35, 35, 30, 30, 30 pixels respectively for the experimentin Figure 4. Each image sequence contains 100 frames. To evalu-ate the accuracy of the single-marker approach and our method, wecalculated the reprojection error of real dots onto the correspond-ing camera images. Specifically, we measured the residual sum-squared error (in pixel) of inlying dot matches, and normalized itby the number of inlying matches to get the reprojection error. Fig-ure 4 shows the reprojection error as a function of number of dotsper basic marker. It can be seen that increasing the number of dotsonly slightly decreases the reprojection error in both methods. In

Figure 4: Localization accuracy measured as the reprojection errorof real dots positions onto the camera image as a function of numberof dots per basic marker.

Figure 5: Comparing localization accuracy of our method and AR-ToolKitPLus in terms of reprojection pixel error for different numberof feature points in the camera view.general, the single-marker approach uses more dots for localizationthan ours since in our method some dots are not grouped into anymarker thus not used to form any compound marker. Thus localiza-tion is theoretically more accurate with the single-marker approach.However, Figure 4 shows that the difference in the reprojection er-ror of the two methods is only about 0.5 pixel for all the tested con-figurations. In summary, our method achieves comparable accuracyto the single-marker approach but has the significant advantage ofscalability in terms of memory usage.

We also compared the localization accuracy of our map of ran-dom dot markers with a map of ARToolKitPlus’s ID-coded mark-ers, again using the synthetic data. For fair comparison, we en-sured that the numbers of feature points (dots in our method versusmarker corners in the ALToolKitPlus map) from camera views areroughly the same for the two methods. It can be seen from Figure 5that our method is more accurate and stable than the ARToolKitPlusmarker map.4.2 Latency

We tested our system on an Sumsung Galaxy Note 10.1 tablet.Our system is able to run at interactive frame rate. The detailedtimings involved in the intermediate steps can be found in Table 1.

5 APPLICATION

To demonstrate the performance of our platform, we implementeda Bomberman game on Android phones (see the accompanyingvideo), supporting up to four players playing together. Facing themobile cameras downwards, players physically move inside themap to control the movement of the bombermen inside the virtualworld, which is a 12 by 10 world consisting of solid walls, mon-sters, bombs and fires. The bomberman is vulnerable to both mon-

Tasks Time (ms)Extract dot centers from camera image < 20Calculate LLAH descriptors < 10Detect basic markers by LLAH matching < 30Form compound markers and estimate location < 5

Table 1: Timings for the intermediate steps of our technique

3

To appear in IEEE Virtual Reality 2013 proceedings

Figure 6: Photo of gameplay on Android phones

ster and fire, but it has the ability to set up a time bomb to releasefire after a certain period. The monsters move randomly, and dieimmediately upon touching the fire. The solid wall is set at regulargrid positions to restrict the movement of the bombermen and mon-sters and the spread of the fire. The trick to win the game is to avoidtouching monsters at all time, set the bomb at monster’s next pos-sible path and move out of the fire range as soon as possible. Thedimension of the map used in this game is 6 by 10 and each basicmarker is of size 80cm x 40cm, so the total gaming area covered is4.8m x 4m, which is large enough to cover an interacting space for4 people.

6 DISCUSSIONS

6.1 GameplayAn interesting design issue arises for the proposed middle-scale

platform is how to force the player in the physical environment tomove only in valid areas presented in the virtual world. For exam-ple, in the Bomberman game described above, there is no physicalobstacle on the marker map to prevent players from stepping into re-gions representing the solid walls in the virtual world. Our solutionis to apply certain penalty such as immobilizing the bomberman ifthe current location is inside the walls, making it unable to duck themonsters, until the player moves back to the rooted location and thebomberman is free to move to the next valid position again.6.2 Marker map installation

A potential difficulty of using our platform is how to deploy themarker map in an actual environment. In our prototype, we print themarkers on A3 papers and assemble the markers for demonstrationpurpose. The random dot marker itself enables flexible marker de-sign, where dots can be changed to more pleasant shapes with care-fully chosen colors such that the centroid of each connected compo-nent remains unchanged after binarizing the grey scale counterpart.In this way, the map could be installed in an unobtrusive way, e.g.,as a permanent floor decoration, carpet texture, puzzle pieces etc.

7 CONCLUSION AND FUTURE WORK

This paper presented a scalable map of random dot markers,which enables efficient localization in middle-scale locative mo-biles games. The scalability is achieved by a smart arrangementof repeated random dot markers. In our current system, markersare placed only on the ground surface. Since the camera pose esti-mation method we adopted [15] is applicable to arbitrary multipleplanes, it is interesting to experiment with using vertical walls oreven ceilings to mount the localization maps. We are also inter-ested in experimenting different kinds of games on our platform. Inaddition, we plan to test our system in larger indoor/outdoor areas,e.g., a library, to examine the maximum scalability of our platform.

Finally, it would be interesting to incorporate various display sys-tems, e.g., head-mounted display (HMD) or projectors to providedifferent immersive experience.

ACKNOWLEDGEMENTS

We thank Pengfei Xu, Jingbo Liu and Xiaoguang Han for theirhelp in recording the accompanying video, and the reviewers fortheir constructive comments. This work was partially supportedby grants from the Research Grants Council of HKSAR (No.CityU113610, No. GRF619611), and the City University of HongKong (No. 7002925 and No. 7002776).

REFERENCES

[1] E. Barba and B. MacIntyre. A scale model of mixed reality. InProceedings of the 8th ACM conference on Creativity and cognition,pages 117–126, 2011.

[2] S. Benford, A. Crabtree, M. Flintham, A. Drozd, R. Anastasi, M. Pax-ton, N. Tandavanitj, M. Adams, and J. Row-Farr. Can you see menow? ACM Trans. Comput.-Hum. Interact., 13(1):100–133, Mar.2006.

[3] A. Cheok, F. S. Wan, X. Yang, W. Weihua, L. M. Huang,M. Billinghurst, and H. Kato. Game-city: a ubiquitous large areamulti-interface mixed reality game space for wearable computers. InSymposium on Wearable Computers, pages 156 – 157, 2002.

[4] M. Fiala. Artag, a fiducial marker system using digital techniques. InCVPR, volume 2, pages 590 – 596 vol. 2, june 2005.

[5] M. Fiala and G. Roth. Magic lens augmented reality: table-top andaugmentorium. In ACM SIGGRAPH 2007 posters. ACM, 2007.

[6] A. Henrysson, M. Billinghurst, and M. Ollila. Face to face collabora-tive ar on mobile phones. In ISMAR, pages 80 – 89, 2005.

[7] H. Kato and M. Billinghurst. Marker tracking and hmd calibrationfor a video-based augmented reality conferencing system. In Interna-tional Workshop on Augmented Reality, pages 85 –94, 1999.

[8] G. Klein and D. Murray. Parallel tracking and mapping on a cameraphone. In Mixed and Augmented Reality, 2009. ISMAR 2009. 8th IEEEInternational Symposium on, pages 83 –86, oct. 2009.

[9] M. Klopschitz, G. Schall, D. Schmalstieg, and G. Reitmayr. Visualtracking for augmented reality. In Indoor Positioning and Indoor Nav-igation, pages 1–4, 2010.

[10] T. Nakai, K. Kise, and M. Iwamura. Use of affine invariants in lo-cally likely arrangement hashing for camera-based document imageretrieval. In H. Bunke and A. Spitz, editors, Document Analysis Sys-tems VII, volume 3872 of Lecture Notes in Computer Science, pages541–552. Springer Berlin / Heidelberg, 2006.

[11] O. Oda, L. J. Lister, S. White, and S. Feiner. Developing an augmentedreality racing game. In INTETAIN ’08, pages 2:1–2:8, 2007.

[12] M. Sperber, M. Klinkigt, K. Kise, M. Iwamura, B. Adrian, andA. Dengel. Handwriting reconstruction for a camera pen using ran-dom dot patterns. In ICFHR, pages 160 –165, 2010.

[13] B. Thomas, B. Close, J. Donoghue, J. Squires, P. De Bondi, M. Morris,and W. Piekarski. Arquake: An outdoor/indoor augmented realityfirst person application. In Symposium on Wearable Computers, pages139–146, 2000.

[14] H. Uchiyama and H. Saito. Random dot markers. In Virtual RealityConference (VR), 2011 IEEE, pages 35 –38, march 2011.

[15] Y. Uematsu and H. Saito. Image-based augmentation of virtual objectfor handy camera video sequence using arbitrary multiple planes. InICIP 2005, volume 1, pages I – 1017–20, sept. 2005.

[16] D. Wagner, T. Pintaric, and D. Schmalstieg. The invisible train: acollaborative handheld augmented reality demonstrator. In ACM SIG-GRAPH 2004 Emerging technologies, 2004.

[17] Y. Xu, M. Gandy, S. Deen, B. Schrank, K. Spreen, M. Gorbsky,T. White, E. Barba, I. Radu, J. Bolter, and B. MacIntyre. Bragfish:exploring physical and social interaction in co-located handheld aug-mented reality games. In ACE ’08, pages 276–283, 2008.

[18] Y. Xu, S. Mendenhall, V. Ha, I. Radu, and B. MacIntyre. Trad-offs fordesigning handheld augmented reality game interfaces. In CSCW ’12.ACM, 2012.

4