Egocentric Visitors Localization in Natural Sites · We compare methods based on GPS and vision on the vis-itor localization task in a natural site. Our experiments show that image-based

Journal of Visual Communication and Image Representation (2019)

Contents lists available at ScienceDirect

Journal of Visual Communication andImage Representation

journal homepage: www.elsevier.com/locate/jvci

Egocentric Visitors Localization in Natural Sites

Filippo L. M. Milottaa,1, Antonino Furnaria,1, Sebastiano Battiatoa, Giovanni Signorellob, Giovanni M. Farinellaa,b,1,∗

aUniversity of Catania, Department of Mathematics and Computer Science, Via Santa Sofia - 64, Catania 95125, ItalybUniversity of Catania, CUTGANA, Via Santa Sofia 98, Catania 95123, Italy

A R T I C L E I N F O

Article history:

Egocentric (First Person) Vision, Local-ization, GPS, Multimodal Data Fusion

A B S T R A C T

Localizing visitors in natural environments is challenging due to the unavailability ofpre-installed cameras or other infrastructure such as WiFi networks. We propose toperform localization using egocentric images collected from the visitor’s point of viewwith a wearable camera. Localization can be useful to provide services to both thevisitors (e.g., showing where they are or what to see next) and to the site manager (e.g.,to understand what the visitors pay more attention to and what they miss during theirvisits). We collected and publicly released a dataset of egocentric videos asking 12subjects to freely visit a natural site. Along with video, we collected GPS locationsby means of a smartphone. Experiments comparing localization methods based onGPS and images highlight that image-based localization is much more reliable in theconsidered domain and small improvements can be achieved by combining GPS- andimage-based predictions using late fusion.

c© 2019 Elsevier B. V. All rights reserved.

1. Introduction and Motivation

Localizing the visitors in natural environments such as parksor gardens can be useful to provide a number of services toboth the visitors and the manager of the natural site. For in-stance, position information can be used to help visitors navi-gate the outdoor environment and augment their visit by pro-viding additional information on what is around. On the otherhand, being able to continuously localizing the visitors allowsto infer valuable behavioral information about them by estimat-ing which paths they follow, where they spend more time, whatthey pay attention to, and what elements do not attract the de-sired attention. The automatic acquisition of such behavioralinformation (typically manually collected by means of ques-tionnaires) would be of great importance for the manager of thesite to assess and improve the quality of the provided services.While localization is generally addressed in outdoor environ-

∗Corresponding author: Tel.: +39 095 7337 219; fax: +39 095 330094;e-mail: [email protected] (Giovanni M. Farinella)

1These authors are co-first authors and contributed equally to this work.

ments using GPS, we found that this technology is not alwaysas accurate as needed to address the considered task. Therefore,we investigate the use of computer vision to reliably assess thelocation of visitors in natural outdoor environments. Specifi-cally, we consider a scenario in which the visitors of the site areequipped with a wearable device which is equipped with a cam-era and optionally a screen to provide feedback and augmentthe visit. The wearable device allows to effortlessly collect vi-sual information of the location in which the visitors move andwhat they pay attention to. Moreover, the integration of local-ization algorithms in devices capable of deploying AugmentedReality applications would allow the implementation of usefulinterfaces to provide the visitor with navigation and recommen-dation services.

While the use of wearable and visual technologies in natu-ral sites has an appealing potential, their exploitation has beenlimited by lack of public datasets suitable to design, developand test localization algorithms in the considered context. Tothis aim, we gathered a new dataset of egocentric videos inthe botanical garden of the University of Catania (Fig. 1). Thedataset has been collected by 12 subjects who where asked to

http://www.sciencedirect.com

http://www.elsevier.com/locate/jvci

2 Filippo L. M. Milotta et al. / Journal of Visual Communication and Image Representation (2019)

(a) (b) (c)

Fig. 1. (a) The entrance of the botanical garden of the University of Catania, (b) the general and (c) Sicilian gardens.

visit the site while wearing an egocentric camera. To assess theexploitability of classic data for localization in the consideredcontext, GPS locations have been also acquired during the vis-its using a smartphone, and later synced to the collected videos.Under the guidance of botanic experts, the chosen natural sitehas been divided into 9 contexts and 9 subcontexts, organized ina hierarchical fashion. The dataset has been labeled with boththe area in which the visitor was located at the moment of acqui-sition and the related GPS coordinates. To encourage researchon the considered problem, we publicly release our dataset atthe following URL: http://iplab.dmi.unict.it/EgoNature.

Relying on the collected data, we perform experiments to in-vestigate the exploitation of GPS and vision for visitor localiza-tion in the considered site. We propose to address localizationas a classification task, i.e., the task of determining in whicharea of the site the visitor is currently located. We first study theexploitation of GPS by assessing the performance of standardclassifiers such as Decision Classification Tree (DCT), Sup-port Vector Machines (SVM) and K-Nearest Neighbor (KNN).The exploitation of vision-based technologies to address thetask is investigated by testing the use of three ConvolutionalNeural Networks (CNN) as image classifiers, namely, AlexNet,VGG16 and SqueezeNet. We note that GPS and vision gener-ally allow to achieve complementary results. Hence, we inves-tigate how classification accuracy can be improved by fusingGPS- and vision-based predictions.

Our analysis outlines that: 1) GPS alone does not allow toobtain satisfactory results on the considered dataset, 2) vision-based approaches (CNNs) largely outperform those based onGPS, 3) small improvements can be obtained by fusing GPS-and vision-based approaches, which suggests that the twomodalities tend to leverage, up to a certain extent, complemen-tary information about the location of the visitors. This latterfinding opens the possibility for further research in the joint ex-ploitation of GPS and vision to address localization in naturalenvironments.

In sum, the contributions of this work are as follows:

• We propose a dataset of egocentric videos collected in anatural site for the purpose of visitor localization. Thedataset has been collected by 12 subjects, contains about6 hours of recording, and is labeled for the visitor local-ization task. To our knowledge this dataset is the first of

its kind and we hope that it can be valuable to the researchcommunity;

• We compare methods based on GPS and vision on the vis-itor localization task in a natural site. Our experimentsshow that image-based approaches are accurate enough toaddress the considered task, while GPS-based approachestend to achieve reasonable results;

• We investigate the benefits of fusing GPS and vision to im-prove localization accuracy. Specifically, our experimentssuggest that better results can be obtained by fusing thetwo modalities, which encourages further research in thisdirection.

This paper extends the work presented in [1]. Specifically,in this paper, we publicly release the proposed EgoNaturedataset along with localization labels and paired GPS coor-dinates. Additionally, we study localization with respect tothree granularity levels: contexts, sub-contexts and the unionof the two, whereas in [1] only the context localization levelwas considered. The experimental evaluation has also been ex-tended considering a 3-fold validation scheme (whereas a singletrain/test split was considered in [1]) and including comparisonswith more image- and GPS-based methods such as AlexNet,VGG16, KNN and SVM.

The remainder of the paper is structured as follows: the re-lated works are discussed in Section 2. Section 3 describes theproposed dataset. The compared localization approaches arediscussed in Section 4. Experimental results are given in Sec-tion 5. Section 6 concludes the paper and discusses possiblefuture works.

2. Related Works

Our work is related to different research topics, includingwearable computing and egography, the exploitation of com-puter vision in natural environments, localization based onwireless and BLE devices, image-based localization, as well aslocalization based on a joint exploitation of images and GPS. Inthe following sections, we revise relevant works belonging tothe aforementioned research lines.

Filippo L. M. Milotta et al. / Journal of Visual Communication and Image Representation (2019) 3

2.1. Wearable Computing and Egography

Our work is related to previous investigations on wearablecomputing [2] and egography [3]. Among the pioneering workson the topic, Mann et al. [4] first introduced the concept ofwearable computing, including the processing of images andvideos collected from a first-person view. Subsequent worksmainly concentrated on topics related to augmented reality andlocation recognition. Mann and Picard proposed Video Or-bits [5], a featureless algorithm to register pairs of images.Mann and Fung [6] further explored the use of Video Orbitsto obtain mediated reality on an eyewear devices. Starner etal. [7] investigated algorithms for location and task recognitionfrom wearable cameras in the context of the game of Patrol.Jebara et al. [8] presented “DyPERS”, a werable system us-ing augmented reality to automonously retrieve “media mem-ories” based on association with real objects detected by thesystem. Torralba et al. [9] contributed a system to performlocation and scene recognition in a known environment fromwearable cameras. Spriggs et al. [10] investigated the problemof recognizing the actions performed by the camera wearer totemporally segment egocentric video. Kitani [11] proposed analgorithm for automatic real-time video segmentation of first-person sports videos. Lee et al. [12] investigated the role ofpeople and objects for egocentric video summarization. Lu andGrauman [13] proposed an approach for story-driven summa-rization of egocentric videos. Karaman et al. [14] investigatedthe use of wearable devices for assistive technologies. Rhine-hart et al. [15] proposed a system for future activity forecastingwith inverse reinforcement learning from first-person videos.Furnari et al. [16] investigated algorithms to anticipate user-object interactions from first-person videos.

Many works on wearable computing considered localizationas a useful component to develop more sophisticated applica-tions. However, most of them [7, 9, 15] have generally con-sidered indoor environments. Differently from these works, weexplore the task of localizing the camera wearer in natural out-door contexts. Recent works have shown that ConvolutionalNeural Networks can be successfully used to localize a cam-era both at the levels of location recognition and camera poseestimation [17, 18, 19, 20]. Coherently with these works, inthis paper, we benchmark several approaches based on Convo-lutional Neural Networks on the proposed dataset.

2.2. Exploitation of Computer Vision in Natural Environments

This line of works mainly investigated classification prob-lems involving plants and trees. Among these works, Kumaret al. [21] proposed Leafsnap, a computer vision system forleaf recognition able to identify 184 species of trees from theNorth-eastern United States. The proposed system is integratedin a mobile application allowing users to take photos of leavesplaced on a white sheet to segment them and remove stems.The leaf silhouettes are represented using histograms of cur-vature on multiple scales. Leaves are hence identified witha Nearest Neighbor search by considering an intersection dis-tance measure. Wegner et al. [22] designed a framework forthe recognition of trees in a urban context. The frameworkmatches aerial images of trees from Google maps with street

view images. This process allows to localize the position oftrees on public street sides. Along with the work, the authors re-leased the Pasadena Urban Trees dataset, which contains about100,000 images of 80,000 trees labeled according to speciesand locations. Van Horn et al. [23] collected the iNat2017dataset employing the “iNaturalist” expert network2. The net-work allows naturalists to share photos of biodiversity acrossthe world. iNat2017 contains images of about 8,000 species ac-quired in natural places. The species are characterized by highvisual variability, high similarity among species and a largenumber of imbalanced and fine-grained categories. A challengeon this dataset has been proposed by the authors to encourageresearch on the field. Joly et al. [24] proposed the LifeCLEFdataset and four related challenges: audio-based bird identifica-tion, image-based plant identification, vision-based monitoringof sea-related organisms, and location-based recommendationof species.

The aforementioned works have mainly proposed datasetsand computer vision algorithms for classification of plants andtrees acquired by a third person camera. Differently, we con-sider the problem of localizing the visitors of a natural site bymeans of images acquired by a first person point of view.

2.3. Localization based on Wireless and BLE Devices

Previous works have investigated the localization problemusing mobile wireless devices [25], and bluetooth low energy(BLE) [26, 27]. In particular, Alahi et al. [25] designed amethod to improve GPS human localization using a combina-tion of fixed antennas, RGB cameras and mobile wireless de-vices such as smartphones and beacons. The authors used amulti-modal approach in which RGB visual data is analyzedjointly with wireless signals. Signal trilateration and propaga-tion models are used together with tracking methods in the RGBdomain to localize people in indoor environments. Ishihara etal. [26] have studied how people localization can be performedwith a beacon-guided approach. This requires the installationof Bluetooth Low Energy (BLE) signals emitters in the envi-ronemnt. The authors designed a method in which localizationbased on radio waves is combined with Structure from Motion(SfM) performed from visual data. However, as stated by theauthors, SfM is still a challenging task in real outdoor scenar-ios, due to the presence of little to no distinctive visual features(i.e., consider a garden with many plants as in our case). Theauthors proposed an improvement of the approach in [27]. Inthis follow-up, inference machines have been trained on pre-viously collected pedestrian paths to perform user localization,hence reducing position and orientation error.

While the exploitation of Wireless and BLE devices can beconvenient in certain indoor scenarios, it is not generally thecase in large natural outdoor environments. This is mainly dueto the lack of existing infrastructures such as WiFi networksand to the difficulties of installing specific hardware across thesites. Differently from the aforementioned works, in this paperwe investigate the exploitation of ego-vision and GPS signals,

2https://www.inaturalist.org/ (accessed 15-Jan-2019)

https://www.inaturalist.org/


which do not require to install specific hardware in the naturalsite.

2.4. Image-Based LocalizationPrevious works have studied image-based localization as a

classification problem. Furnari et al. [17] addressed the prob-lem of recognizing user-specified personal locations from ego-centric videos. In this context, localization is addressed asan “open-set” classification problem, where the system shouldidentify the pre-specified locations and reject those which havenot been specified at training time. Starner et al. [7] investigatedthe use of wearable cameras to localize users from egocentricimages. Localization is addressed as a “closed-set” room-levelclassification problem as the user is assumed to move withina limited set of rooms in a building. Santarcangelo et al. [18]studied the exploitation of multimodal signals collected fromshopping carts to localize the customers of a retail store. Local-ization is addressed as a classification problem in which eachimage collected by the shopping cart is associated to the corre-sponding department in the retail store. The location informa-tion is hence exploited for behavioral understanding in a “Vi-sual Market Basket Analysis” scenario. Ragusa et al. [28] con-sidered the problem of localizing the visitors of a cultural sitefrom egocentric video. Localization is addressed as a classifi-cation task to understand in which room the visitor is locatedfrom egocentric images. In the considered domain, localizationinformation can be used by museum curators and site managersto improve the arrangement of their collections and increase theinterest of their audience. Differently from the proposed work,Ragusa et al. [28] address visitors’ localization in indoor envi-ronments, where GPS coordinates cannot be reliably used. Bet-tadapura et al. [29] leveraged a reference dataset for reducingthe open-ended data-analysis problem of image classificationinto a more practical data-matching problem. The idea of char-acterizing locations through features is present also in the workof Kapidis et al. [30]. In the considered indoor scenario, thefeatures are related to the presence/absence of specific objects.Indeed, detected objects can be used as discriminant descriptorsfor a set of images taken from the same room.

Similarly to the aforementioned works, we tackle localiza-tion as a classification problem. This is done by dividing theconsidered space into non-overlapping areas. Differently fromthe discussed works, however, we study the the problem in thecontext of natural sites and investigate the exploitation of GPSand ego-vision to address the considered localization task. Wewould like to note that, while dividing an indoor space into co-herent areas is straightforward (e.g., rooms are usually consid-ered as areas), defining meaningful areas in an outdoor contextis a less well-defined problem. In this work, we followed the ad-vice of botanic experts in order to divide the considered spacein areas meaningful for the visitors. Indeed, the different ar-eas contain different plants belonging to different families andhence estimating the location of visitors with respect to theseareas can provide useful information on their behavior.

2.5. Image-Based Geo-LocalizationOur research is also related to previous works on image-

based geo-localization, which consists in inferring geo-location

(e.g., GPS coordinates) from images. Hays and Efros [31] col-lected a dataset of 6 million GPS-tagged images from the Inter-net and proposed a method capable of estimating geo-locationfrom a single image by predicting a probability distribution overgeographic locations. Zamir and Shah [32] performed image-based geo-localization by indexing in a tree SIFT keypointsextracted from the training images. At test time, the tree isqueried using the SIFT descriptors of keypoints detected in aquery image. A GPS-tag-based pruning method is used to dis-card less reliable descriptors. Lin et al. [33] introduced a cross-view feature translation approach which allows to learn the re-lationship between ground level appearance and over-head ap-pearance. This approach allows to leverage over-head maps toperform geo-localization also in regions containing few geo-tagged ground level images. Zamir and Shah [34] presenteda nearest neighbor feature matching method based on general-ized minimum clique graphs for image-based geo-localization.Weyand et al. [35] proposed a deep network which can localizeimages of places exploiting different cues such as landmarks,weather patterns, vegetation, road markings, or architectural de-tails. Zemene et al. [36] tackled the problem of geo-localizationof city-wide images, addressing the task by clustering local fea-tures in the images. Features from reference images (i.e., archi-tectural details, road markings, and characteristic vegetation)are hence used for Dominant Set Clustering (DSC), which canbe seen as an improvement of Nearest Neighbors based meth-ods [32, 34].

Differently from these works, we focus on image-based lo-cation recognition at the context level, which already allows toobtain useful insights on the behavior of visitors. As shown inthe experiments, this is also due to the fact that GPS locationscollected in the considered domain are not reliable enough to beconsidered a valid ground truth to train geo-localization meth-ods.

2.6. Joint Exploitation of Images and GPS for Localization

The combination of GPS and visual information to localizeusers in an environment has been investigated in the past. Capiet al. [37] proposed an assistive robotic system able to guidevisually impaired people in urban environments. Navigation isobtained with a multimodal approach by combining GPS, com-pass, laser range finders, and visual information to train neuralnetworks. The system has been validated in a controlled en-vironment, but the authors have shown that it can also adaptto environment changes. NavCog [38] is a smartphone-basedsystem which performs accurate and real-time localization overlarge spaces.

Similarly to these works, we investigate how to combineGPS and vision. However, differently from previous works, ourstudy focuses on ego-vision in the context of natural environ-ments.

3. Dataset

To study the localization of visitors in natural sites, we gath-ered a dataset of egocentric videos in the botanical garden of


Fig. 2. Top: map of the Botanical Garden with contexts (blue numbers) and subcontexts (red numbers). Bottom-left: list of the contexts. Bottom-right: listof subcontexts.

the University of Catania3(Fig. 1(a)). The botanical garden ismade up of two main parts: the General Garden (Orto Generaleor Orto Universale, Fig. 1(b)) and the Sicilian Garden (OrtoSiculo, Fig. 1(c)). The site hosts several plant species and hasan extension of about 16,000 m2. Under the guidance of botanic

3http://ortobotanico.unict.it/

experts, the garden has been divided into 9 main contexts whichidentify meaningful areas for the visitors of the natural site. Forinstance, some examples of contexts are “entrance”, “green-house”, and “central garden”. We would like to highlight thatthe considered botanic garden has been divided into contextsconsidering areas which are useful to characterize the behav-ior of visitors and understand their interests. Indeed, a context


Fig. 3. Alignment process between GPS measurements and video frames.GPS and video data are acquired at different rates, hence several videoframes are aligned with the same GPS measurement. Black dotted linesin the figure represent the boundaries of time-slot defined by each GPSacquisition. Boundaries are defined exactly in the middle temporal pointbetween two GPS acquisitions (i.e., at distance D1 for the first and secondGPS measurements, at distance D2 for the second and third GPS measure-ments, etc.).

contains plants organized by their families. The contexts of thegarden in which the visitors have to be localized are highlightedwith blue numbers in Fig. 2(top). The table in Fig. 2(bottom-left) lists the 9 contexts and reports some sample images thereincaptured. Note how, while some of the contexts tend to be char-acterized by distinctive features (e.g., compare the sample im-ages from context 3 “greenhouse” with the sample images fromcontext 2 “monumental building”), the contexts are, in general,characterized by low inter-class variability due to the predom-inance of plants and to the presence of similar architectural el-ements. To allow for a more fine-grained localization of thevisitors, context 5 “Sicilian Garden” has been further dividedinto 9 subcontexts by partitioning the Sicilian Garden wheredifferent Sicilian species are hosted. These additional subcon-texts are highlighted with red numbers in Fig. 2(top) and listedin Fig. 2(bottom-right).

We collected a set of egocentric videos within the botani-cal garden asking 12 volunteers to freely visit the natural sitewhile wearing a head-mounted camera and a GPS receiver. Thedata have been collected in different days and contain weathervariability. The wearable camera, a Pupil Mobile Eye Track-ing Headset4, was used to collect videos of the visits from thepoints of view of the visitors. A Honor 9 smartphone was usedto collect GPS locations by means of the MyTracks Androidapplication5. The video camera and the GPS device collecteddata at different frame-rates. It should be noted that this sam-pling scenario is common when dealing with multi-modal datacollected using different sensors, as investigated in previous re-search [39]. In this work, in order to assign a GPS positionto each video frame, we performed an alignment procedure inwhich each video frame was associated with the closest GPSmeasurement in time, as illustrated in Fig. 3. Please note thatthis procedure allows for the replication of GPS positions alongdifferent contiguous video frames. All videos have been vi-sually inspected and each frame has been labeled to specifyboth the context and subcontext in which the visitor was op-erating. The dataset contains labeled videos paired with frame-

4https://pupil-labs.com/pupil/5https://play.google.com/store/apps/details?id=com.zihua.android.mytracks&hl

Table 1. Classification granularities used to study the problem of visitor lo-calization on the collected dataset. The indices of contexts and subcontextsare coherent with the ones reported in Fig. 2.

Name Granularity (Sub)Contexts9 Contexts Coarse 1, 2, 3, 4, 5, 6, 7, 8, 9

9 Subcontexts Fine 5.1, 5.2, 5.3, 5.4, 5.5,5.6, 5.7, 5.8, 5.9

17 Contexts Mixed 1, 2, 3, 4, 5.1, 5.2,5.3, 5.4, 5.5, 5.6, 5.7,5.8, 5.9, 6, 7, 8, 9

Table 2. The three folds in which the dataset has been divided for evaluationpurposes.

Fold Subjects ID Frames1 2, 3, 7, 8 23,1452 0, 5, 6, 9 14,6593 1, 4, 10, 11 25,777

wise GPS information for a total of 6 hours of recording, fromwhich we extract a large subset of 63,581 frames. The reader isreferred to Fig. 2(bottom) for visual examples from the dataset.

The collected dataset can be used to address the problem oflocalizing the visitors of a natural site as a classification task.Specifically, since each frame of the dataset has been labeledwith respect to both contexts and subcontexts, location-basedclassification can be addressed at different levels of granularity,by considering 1) only the 9 contexts (coarse localization), 2)only the 9 subcontexts (fine localization), or 3) the 17 contextsobtained by considering the 9 contexts and substituting context5 “Sicilian garden” with its 9 subcontexts (mixed granularity).Please note that when we perform experiments on the 9 sub-contexts, only a subset of 15,750 frames labeled as “Context 5”is considered. Table 1 summarizes the classes involved whenconsidering each of these classification schemes.

For evaluation purposes, the dataset has been divided into 3different folds by splitting the set of videos from the 12 vol-unteers into 3 disjoint groups, each containing videos from 4subjects. We divided the set of videos such that the frames ofeach class are equally distributed in the different folds. Table 2reports information on which videos acquired by the differentsubjects have been considered in each fold, and the number offrames in each fold.

To the best of our knowledge, this is the first largedataset for localization purposes in natural sites domain.The dataset is publicly available at the following URL:http://iplab.dmi.unict.it/EgoNature.

4. Methods

As previously discussed, we consider localizing the visitorof the natural site as a classification problem. In particular, weinvestigate classification approaches based on GPS and images,as well as methods jointly exploiting both modalities. Each ofthe considered methods is trained and evaluated on the proposedEgoNature dataset according to the three different levels of lo-calization granularity reported in Table 1.


4.1. Localization Methods Based on GPSWhen GPS coordinates are available, outdoor localization is

generally performed by directly analyzing such data. The mainassumption behind this approach is that GPS coordinates can beretrieved accurately, in which case vision would not be neededto perform location recognition. As we will show in the ex-periments, however, accurate estimation of GPS coordinates isnot trivial in the considered context, also due to the presence oftrees covering the sky. To investigate localization approachesbased only on GPS, we consider different popular classi-fiers. Specifically, we employed Decision Classification Trees(DCT) [40], Support Vector Machines (SVM) [41] with linearand RBF kernels and k-nearest neighbor (KNN) [42]. Each ofthe considered approaches takes the raw GPS coordinates (x, y)as input and produces as output a probability distribution overthe considered classes, conditioned on the input, where c repre-sents the contexts to be predicted. We tune the hyperparametersof the classifiers performing a grid search with cross valida-tion. In particular, we optimize: the maximum depth of thedecision tree for DCT (search range: {1, 2, . . . , 100}), the val-ues of C (search range: {1, 10, 100, 1000}), γ (search range:{0.01, 0.05, 0.001, 0.0001, auto}), and kernel (i.e., linear andrbf) for SVMs, and the number of neighbors k for KNN classi-fiers (search range: {1, 2, . . . , 100}). We remove duplicate GPSmeasurements from the training set during the learning phaseof GPS-based methods, as we noted that this improved perfor-mance. Please note that, for fair comparison, duplicate mea-surements are not removed at test time.

4.2. Localization Methods Based on ImagesImage-based localization methods assume that no GPS co-

ordinates are available and hence address location recognitionby directly processing images captured from the wearable de-vice. Despite performing image-based localization is not trivial,the main advantage of this approach is that image-based sensingcan be less noisy than estimating GPS coordinates. Each image-based localization method takes as input an image I and pre-dicts a probability distribution over the contexts p(c|I). To studyimage-based localization, we consider three different CNN ar-chitectures: AlexNet [43], Squeezenet [44] and VGG16 [45].The three architectures achieve different performances on theImageNet classification task [46] and require different compu-tational budgets. In particular, AlexNet and VGG16 requirethe use of a GPU at inference time, whereas SqueezeNet is ex-tremely lightweight and has been designed to run on CPU (i.e.,can be exploited on mobile and wearable devices). We initial-ize each network with ImageNet-pretrained weights and fine-tune each architecture for the considered classification tasks.The CNNs are trained with Stochastic Gradient Descent for 300epochs with a learning rate equal to 0.001. The batch size wasset to 512 in the case of SqueezeNet, 256 in the case of AlexNetand 64 in the case of VGG16. To reduce the effect of overfit-ting when comparing different models, at the end of training weselected the epoch achieving the best results on the test set.

4.3. Localization Methods Exploiting Jointly Images and GPSWhile GPS- and Image-based methods assume no other in-

formation to be available, fusing the two sources of information

Table 3. Performances of localization methods based on GPS and imagesin terms of Accuracy%.

FoldsMethods 1 2 3 Average

9C

onte

xts

DCT 78.76 46.83 76.53 67.37SVM 78.89 52.59 78.50 69.99KNN 80.38 54.34 81.05 71.92

AlexNet 90.99 90.72 89.63 90.45SqueezeNet 91.24 93.08 91.40 91.91

VGG16 94.26 95.59 94.08 94.64

9Su

bcon

text

s DCT 55.26 40.47 38.63 44.78SVM 52.60 43.03 49.85 48.49KNN 59.71 43.58 51.61 51.63


VGG16 90.68 91.47 87.18 89.78

17C

onte

xts DCT 65.03 37.25 64.95 55.74

SVM 66.71 42.10 64.62 57.81KNN 67.50 43.84 65.75 59.03


VGG16 91.35 94.03 91.10 92.16

can lead to improved performance. Indeed, even if GPS mea-surements can be inaccurate, they generally provide a roughestimate for the position of the visitor. On the contrary, image-based localization can provide more accurate localization at thecost of a higher variance. For instance, two different contextscan present images with similar visual content, which can leadto erroneous predictions (see Fig. 2). We hence expect littlecorrelation between the mistakes made by approaches based onimages and GPS. To leverage such complimentary nature of thepredictions, we investigate the effect of fusing them by late fu-sion. Specifically, let α be the vector of class scores producedby a CNN for a given visual sample and let βi be the vectorsof scores produced by n different localization approaches basedon GPS. Given a weight w, we fuse the considered predictionsusing the following formula: w · α +

∑ni=0 βi. The final goal of

a localization method exploiting both images and GPS is henceto produce a probability distribution p(c|I, x, y) over contextsconditioned on both the input image and GPS coordinates.

5. Experimental Results

In this section, we report and discuss the results of exper-iments performed on the proposed dataset by considering thedifferent methods discussed in previous section.

5.1. Methods Based Only on Images or GPS

Table 3 reports the results of the methods based on visualand GPS data when used separately. Results are shown in termsof accuracy%, according to the three previously discussed lo-calization granularities (9 Contexts, 9 Subcontexts, 17 Con-texts). All results are computed in a 3-fold validation setting.


Specifically, columns 2 − 4 of Table 3 report the results ob-tained considering one of the folds as test set and the other onesas training sets, whereas the last column reports the averageof the results obtained in each fold. The best average resultamong the image-based methods are reported in bold, whereasthe best average results among the GPS-based methods are un-derlined. As shown by the table, KNN is the best performingmethod among the ones based on GPS, followed by SVM andDCT. This is consistent for all localization levels (9 Contexts,9 Subcontexts and 17 Contexts) and for most of the individualfolds. Analogously, among the image-based classifiers, VGG16is the best performing method, followed by SqueezeNet andAlexNet. This behavior is consistent for all folds and localiza-tion granularities and is probably related to the higher perfor-mances shown by VGG16 on the ImageNet classification chal-lenge [45].

As can be noted from Table 3, the methods based only onGPS do not achieve satisfactory localization performances, es-pecially if compared to those based on CNNs. For instance,KNN, which is the best performing method among those basedon GPS, achieves an average accuracy of only 71.92% on 9Contexts, whereas VGG16 obtains an accuracy of 94.64% onthe same visitor localization problem. The accuracy of themethods based on GPS further drops down when a more pre-cise localization is required in the 9 Subcontexts granularity. Itis worth to note that, while the performances of the methodsbased on GPS drop down by about 20% in the 9 Subcontextslocalization challenge (e.g., KNN score drops from 71.92% to51.63%), the performances of image-based methods drop downonly by a few percentage points to address the same problem(e.g., VGG16 score drops from 94.64% to 89.78%). Interme-diate performances are obtained when considering localizationat the granularity of 17 Contexts, with a localization accuracyof 92.16%. This comparison highlights how GPS informationacquired using a smartphone is not suitable alone for the local-ization of visitors in a natural site, while approaches based onimages achieve much more promising results. It should also benoted that the performance of methods based on GPS might alsobe affected by the lower rate at which GPS coordinates are col-lected, as compared to images, as highlighted in Fig. 3, whichhighlights the higher reliability of image-based localization inthe considered context.

Fig. 4 reports the confusion matrices obtained by the KNNclassifier (i.e., the best performer among the methods based onGPS, according to the different localization granularities). Mostof the errors are obtained when a sample is wrongly classifiedas belonging to a neighboring class (e.g., see confusion between“Con 1” and “Con 2” and between “Con 5” and “Con 6” inFig. 4(a), and by considering the map in Fig. 2). This effectis more pronounced at the finer granularities of 9 Subcontexts(Fig. 4(b)) and 17 Contexts (Fig. 4(c)). This fact confirms thatGPS is not appropriate to solve the task under consideration.

Fig. 5 reports the confusion matrices obtained according tothe different localization granularities by the VGG16 classifier,which is the best performing one among the image-based meth-ods. A comparison between Fig. 4 and Fig. 5 shows the su-perior performances of image-based methods in the considered

Table 4. Late fusion results (accuracy%) related to the 9 Contexts granu-larity level for different values of the late fusion weight w.

CNN weight for Late Fusion (w)1 2 3 4 5

9C

onte

xts

AlexNet + DCT 67.28 81.15 91.33 91.01 90.88AlexNet + SVM 70.38 91.70 91.27 90.99 90.87AlexNet + KNN 72.17 90.90 91.21 90.98 90.87AlexNet + DCT + SVM 67.95 76.49 91.65 91.54 91.31AlexNet + DCT + KNN 67.72 78.48 91.10 91.53 91.28AlexNet + SVM + KNN 70.99 81.54 91.76 91.52 91.23AlexNet + DCT + SVM + KNN 69.18 75.74 85.33 91.76 91.67SqueezeNet + DCT 67.28 82.48 92.47 92.27 92.20SqueezeNet + SVM 70.38 92.96 92.40 92.24 92.16SqueezeNet + KNN 72.17 91.90 92.39 92.24 92.14SqueezeNet + DCT + SVM 67.95 76.80 93.05 92.63 92.43SqueezeNet + DCT + KNN 67.72 78.84 92.12 92.62 92.44SqueezeNet + SVM + KNN 70.99 81.99 93.02 92.58 92.40SqueezeNet + DCT + SVM + KNN 69.18 76.04 86.11 93.07 92.68VGG16 + DCT 67.28 85.33 94.86 94.80 94.76VGG16 + SVM 70.38 95.14 94.86 94.81 94.76VGG16 + KNN 72.17 93.86 94.84 94.79 94.76VGG16 + DCT + SVM 67.95 77.41 95.09 94.95 94.87VGG16 + DCT + KNN 67.72 79.72 94.06 94.95 94.87VGG16 + SVM + KNN 70.99 83.05 95.17 94.94 94.86VGG16 + DCT + SVM + KNN 69.18 76.43 87.34 95.14 94.99Average: AlexNet 69.38 82.29 90.52 91.33 91.16Average: SqueezeNet 69.38 83.00 91.65 92.52 92.35Average: VGG16 69.38 84.42 93.75 94.91 94.84Total Average 69.38 83.23 91.97 92.92 92.78

contexts. In particular, confusion between adjacent classes is amuch more reduced phenomenon than in the case of GPS-basedlocalization.

5.2. Methods Exploiting Jointly Images and GPS (Late Fusion)

Table 4-6 report the results obtained combining approachesbased on images and GPS with late fusion for the three consid-ered localization granularities. Specifically, we investigate allpossible combinations of a CNN with one, two or three GPS-based classifiers for different values of the late fusion weightw ∈ {1, 2, 3, 4, 5}. Best results per row are reported in bold.Best results per CNN are underlined. In sum, the best resultsare obtained when the class scores predicted by the CNN areweighted 4 times more (w = 4) than the scores produced byGPS-based approaches. This further indicates that image-basedlocalization is much more reliable than GPS-based localizationin the considered domain.

Table 7 summarizes the improvements obtained by late fu-sion approach with respect to image-based approaches. Thecomparisons are performed considering the classifiers whichfuse CNNs with DCT, SVM and KNN with a late fusion weightw = 4 (i.e., “4 · Alexnet + DCT+SVM+KNN”, “4 · Squeezenet+ DCT+SVM+KNN”, “4 · VGG16 + DCT+SVM+KNN”). Inaverage, late fusion allows to obtain improvements of +1.25%,+1.03%, and +0.48% in terms of accuracy% over AlexNet,SqueezeNet, and VGG16, respectively. It should be noted thatimprovements are larger for the coarse 9 Contexts granularity,in which even inaccurate GPS positions can be useful.

Fig. 6 reports the confusion matrices obtained with late fu-sion of VGG16, DCT, SVM, and KNN, with respect to thethree localization granularities. As can be observed comparingFig. 5 and Fig. 6, fusing image-based predictions with GPS-based predictions allows to marginally reduce confusion among


(a) 9 Contexts

(b) 9 Subcontexts

(c) 17 Contexts

Fig. 4. Confusion matrices of the KNN classifier based on GPS for the lo-calization granularities of (a) 9 Contexts, (b) 9 Subcontexts, (c) 17 Contexts.Values are reported as percentages.

(a) 9 Contexts

(b) 9 Subcontexts

(c) 17 Contexts

Fig. 5. Confusion matrices of the VGG16 classifier based on images forthe localization granularities of (a) 9 Contexts, (b) 9 Subcontexts, (c) 17Contexts. Values are reported as percentages.


(a) 9 Contexts

(b) 9 Subcontexts

(c) 17 Contexts

Fig. 6. Confusion matrices of the classifier obtained combining with latefusion (w = 4) VGG16, DCT, SVM, and KNN for the localization gran-ularities of (a) 9 Contexts, (b) 9 Subcontexts, (c) 17 Contexts. Values arereported as percentages.

Table 5. Late fusion results (accuracy%) related to the 9 Subcontexts gran-ularity level for different values of the late fusion weight w.


9Su

bcon

text

s


non-adjacent classes in the case of 9 Contexts (compare thenumbers far from the diagonal in Fig. 5(a) with the related onesof Fig. 6(a).

Table 8 reports some examples of images incorrectly classi-fied by VGG16, but correctly classified when using late fusion.The second and third columns of the table report the predictedand expected Context/Subcontext classes. The IDs are coher-ent with the ones reported in Fig. 2. The fourth column reportsimages belonging to the expected class but similar to the inputimage. The examples show that some errors are due to visualambiguities due similar, yet distinct environments.

5.3. Summary of the Best Performing Methods

To summarize the results, Table 9 reports the accuracy of thebest performing methods for each of the considered modali-ties, i.e., GPS, images, and late fusion. GPS-based methodsdo not allow to obtain satisfactory results, especially when afiner-grade localization is required (i.e., in the case of 9 Sub-contexts and 17 Contexts). On the contrary, the VGG16 methodallows to obtain much better results. Interestingly, while theperformances of KNN drop down by about 20% when goingfrom 9 Contexts to 9 Subcontexts classification, for VGG16, weonly observe a drop in performance of less than 5%. Late fu-sion always allows to improve over the results obtained whenusing KNN or VGG16 alone. However, it should be noted thatimprovements over VGG16 are marginal.

Fig. 7 reports the F1 scores computed by the considered bestperforming methods with respect to the three localization gran-ularities. In particular, we report the F1 score related to eachclass, as well as the average of the F1 scores. As shown inFig. 7, KNN achieves low performance in some contexts andsubcontexts (e.g., it achieves 0 for Context 9 in Fig. 7(a) andFig. 7(c) and low values for Subcontexts 5.4 and Subcontexts


Table 6. Late fusion results (accuracy%) related to the 17 Contexts granu-larity level for different values of the late fusion weight w.


17C

onte

xts


Table 7. Improvements in accuracy% obtained using Late Fusion (LF).AlexNet SqueezeNet VGG16

AlexNet LF Imp. SqueezeNet LF Imp. VGG16 LF Imp.9 Contexts 90.45 91.76 1.31 91.91 93.07 1.16 94.64 95.14 0.50

9 Subcontexts 83.09 84.12 1.03 85.41 86.33 0.92 89.78 90.12 0.3417 Contexts 85.74 87.15 1.41 88.81 89.82 1.01 92.16 92.75 0.59Avg Imp. 1.25 1.03 0.48

5.8 Fig. 7(a)). VGG16 obtains much better and less variableresults for all granularities, whereas late fusion allows to occa-sionally improve localization accuracy for some classes.

5.4. Computational Analysis

We performed experiments to understand the computationaleffort needed when employing the different approaches. Ta-ble 10 reports the computational requirements of the localiza-tion based on GPS and images. In particular, for each methodwe report the time needed to process a single image in millisec-onds and the memory required by the model. All times havebeen computed on CPU using a four-cores Intel(R) Xeon(R)CPU E5-2620 v3 @ 2.40GHz, averaging over 1000 predictions.Time and memory requirements have been computed separatelyfor each granularity and fold and then averaged. The tableshows that, despite the low accuracy, the methods based on GPSare extremely efficient both in terms of required memory andtime. Among these methods, DCT and KNN are particularly ef-ficient in terms of memory. On the contrary, methods based onCNNs require more memory and processing time. Specifically,VGG16 is the slowest and heaviest method, while Squeezenet isthe fastest and most compact one. Note that, when late fusion isconsidered, the overhead required to obtain GPS-based is neg-ligible due to the high computational efficiency of GPS-basedmethods.

Table 8. Example of images wrongly classified by CNNs, but correctly clas-sified by Late Fusion approach. Second and third columns show the pre-dicted and expected Context/Subcontext, respectively, using the same IDsshown in Fig. 2.

Input Image Predicted Expected Sample Image from Expected Class

5.3 5.2

5.3 5.9

5.6 2

1 4

Table 9. Accuracy% of the best performing methods of each of the threeconsidered modalities (GPS, Images and Late Fusion).

GPS Images Late Fusion(KNN) (VGG16) (VGG16+DCT+SVM+KNN)

9 Contexts 71.92 94.64 95.149 Subcontexts 51.63 89.78 90.12

17 Contexts 59.03 92.16 92.75

6. Conclusion

We considered the problem of localizing visitors of a naturalsite from egocentric videos. To study the problem, we collecteda new dataset of egocentric videos in the Botanical Garden ofthe University of Catania. Each frame has been labeled witha location-related class and with GPS coordinates which havebeen acquired along with the video using a smartphone. TheEgoNature dataset is publicly released to encourage research inthe field. We performed an experimental evaluation of differentmethods based on GPS, images and the combination of both,according to three localization granularities. The experimentsshow that image-based localization is much more accurate thanGPS-based localization in the considered settings and that fu-sion with GPS-based predictions slightly improve results. Fu-ture works could be devoted to investigate more principle ap-proaches to fuse image and GPS information in order to im-prove localization accuracy and computational efficiency.

Acknowledgments

This research is supported by PON MISE - Horizon 2020,Project VEDI - Vision Exploitation for Data Interpretation,Prog. n. F/050457/02/X32 - CUP: B68I17000800008 - COR:128032, and Piano della Ricerca 2016-2018 linea di Intervento2 of DMI of the University of Catania. The authors would liketo thank Costantino Laureanti for his invaluable help duringdata collection and labelling.


(a) 9 Contexts

(b) 9 Subcontexts

(c) 17 Contexts

Fig. 7. F1-Score for (a) 9 Contexts, (b) 9 Subcontexts, (c) 17 Contexts. Valuesare reported as percentages.

Table 10. Required computational time and memory.Method Time (ms) Mem (MB)

DCT 0.01 0.07SVM 0.01 0.33KNN 0.01 0.12

AlexNet 20.25 217.60SqueezeNet 18.30 2.78

VGG16 434.86 512.32

References

[1] F. L. M. Milotta, A. Furnari, S. Battiato, M. De Salvo, G. Signorello,G. M. Farinella, Visitors localization in natural sites exploiting egovisionand gps, in: 2019 14th International Conference on Computer VisionTheory and Applications (VISAPP), 2019.

[2] S. Mann, S. Mann, Intelligent image processing, IEEE, 2002.[3] S. Mann, K. M. Kitani, Y. J. Lee, M. Ryoo, A. Fathi, An introduction to

the 3rd workshop on egocentric (first-person) vision, in: 2014 IEEE Con-ference on Computer Vision and Pattern Recognition Workshops, IEEE,2014, pp. 827–832.

[4] S. Mann, Wearable computing: A first step toward personal imaging 30(1997) 25–32.

[5] S. Mann, R. W. Picard, Video orbits of the projective group a simpleapproach to featureless estimation of parameters, IEEE Transactions onImage Processing 6 (1997) 1281–1295.

[6] S. Mann, J. Fung, Videoorbits on eye tap devices for deliberately dimin-ished reality or altering the visual perception of rigid planar patches of areal world scene, EYE 3 (2001) P3.

[7] T. Starner, B. Schiele, A. S. Pentland, Visual contextual awareness inwearable computing, in: Proceedings of the International Symposium onWearable Computing, 1998, pp. 50–57.

[8] T. Jebara, B. Schiele, N. Oliver, A. Pentland, Dypers: Dynamic personal

enhanced reality system, in: In Proc. 1998 Image Understanding Work-shop, Citeseer, 1998.

[9] A. Torralba, K. P. Murphy, W. T. Freeman, M. A. Rubin, Context-basedvision system for place and object recognition, in: International Confer-ence on Computer Vision, 2003.

[10] E. H. Spriggs, F. De La Torre, M. Hebert, Temporal segmentation andactivity classification from first-person sensing, in: 2009 IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition Work-shops, IEEE, 2009, pp. 17–24.

[11] K. Kitani, Ego-action analysis for first-person sports videos, IEEE Per-vasive Computing 11 (2012) 92–95.

[12] Y. J. Lee, J. Ghosh, K. Grauman, Discovering important people and ob-jects for egocentric video summarization, in: 2012 IEEE Conference onComputer Vision and Pattern Recognition, IEEE, 2012, pp. 1346–1353.

[13] Z. Lu, K. Grauman, Story-driven summarization for egocentric video,in: Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, 2013, pp. 2714–2721.

[14] S. Karaman, J. Benois-Pineau, V. Dovgalecs, R. Megret, J. Pinquier,R. Andre-Obrecht, Y. Gaestel, J.-F. Dartigues, Hierarchical hiddenmarkov model in detecting activities of daily living in wearable videosfor studies of dementia, Multimedia tools and applications 69 (2014)743–771.

[15] N. Rhinehart, K. M. Kitani, First-person activity forecasting with onlineinverse reinforcement learning, in: Proceedings of the IEEE InternationalConference on Computer Vision, 2017, pp. 3696–3705.

[16] A. Furnari, S. Battiato, K. Grauman, G. M. Farinella, Next-active-objectprediction from egocentric videos, Journal of Visual Communication andImage Representation 49 (2017) 401 – 411.

[17] A. Furnari, G. M. Farinella, S. Battiato, Recognizing personal locationsfrom egocentric videos, IEEE Transactions on Human-Machine Systems47 (2017) 6–18.

[18] V. Santarcangelo, G. M. Farinella, A. Furnari, S. Battiato, Market basketanalysis from egocentric videos, Pattern Recognition Letters 112 (2018)83–90.

[19] A. Kendall, M. Grimes, R. Cipolla, Posenet: A convolutional networkfor real-time 6-dof camera relocalization, in: Proceedings of the IEEEinternational conference on computer vision, 2015, pp. 2938–2946.

[20] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, J. Sivic, Netvlad: Cnnarchitecture for weakly supervised place recognition, in: Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition, 2016,pp. 5297–5307.

[21] N. Kumar, P. N. Belhumeur, A. Biswas, D. W. Jacobs, W. J. Kress, I. C.Lopez, J. V. Soares, Leafsnap: A computer vision system for automaticplant species identification, in: Computer Vision–ECCV 2012, Springer,2012, pp. 502–516.

[22] J. D. Wegner, S. Branson, D. Hall, K. Schindler, P. Perona, Catalogingpublic objects using aerial and street-level images-urban trees, in: Pro-ceedings of the IEEE Conference on Computer Vision and Pattern Recog-nition, 2016, pp. 6014–6023.

[23] G. Van Horn, O. Mac Aodha, Y. Song, A. Shepard, H. Adam, P. Per-ona, S. Belongie, The inaturalist challenge 2017 dataset, arXiv preprintarXiv:1707.06642 (2017).

[24] A. Joly, H. Goeau, H. Glotin, C. Spampinato, P. Bonnet, W.-P. Vellinga,J.-C. Lombardo, R. Planque, S. Palazzo, H. Muller, Lifeclef 2017 laboverview: multimedia species identification challenges, in: InternationalConference of the Cross-Language Evaluation Forum for European Lan-guages, Springer, 2017, pp. 255–274.

[25] A. Alahi, A. Haque, L. Fei-Fei, Rgb-w: When vision meets wireless, in:Proceedings of the IEEE International Conference on Computer Vision,2015, pp. 3289–3297.

[26] T. Ishihara, J. Vongkulbhisal, K. M. Kitani, C. Asakawa, Beacon-guidedstructure from motion for smartphone-based navigation, in: Applicationsof Computer Vision (WACV), 2017 IEEE Winter Conference on, IEEE,2017, pp. 769–777.

[27] T. Ishihara, K. M. Kitani, C. Asakawa, M. Hirose, Inference machines forsupervised bluetooth localization, in: Acoustics, Speech and Signal Pro-cessing (ICASSP), 2017 IEEE International Conference on, IEEE, 2017,pp. 5950–5954.

[28] F. Ragusa, A. Furnari, S. Battiato, G. Signorello, G. M. Farinella, Ego-centric visitors localization in cultural sites, Journal on Computing andCultural Heritage (2019).

[29] V. Bettadapura, I. Essa, C. Pantofaru, Egocentric field-of-view localiza-


tion using first-person point-of-view devices, in: Applications of Com-puter Vision (WACV), 2015 IEEE Winter Conference on, IEEE, 2015, pp.626–633.

[30] G. Kapidis, R. W. Poppe, E. A. van Dam, R. C. Veltkamp, L. P. Noldus,Where am i? comparing cnn and lstm for location classification in egocen-tric videos, in: 2018 IEEE International Conference on Pervasive Com-puting and Communications Workshops (PerCom Workshops), IEEE,2018, pp. 878–883.

[31] J. Hays, A. A. Efros, Im2gps: estimating geographic information from asingle image, in: 2008 ieee conference on computer vision and patternrecognition, IEEE, 2008, pp. 1–8.

[32] A. R. Zamir, M. Shah, Accurate image localization based on google mapsstreet view, in: European Conference on Computer Vision, Springer,2010, pp. 255–268.

[33] T.-Y. Lin, S. Belongie, J. Hays, Cross-view image geolocalization, in:Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, 2013, pp. 891–898.

[34] A. R. Zamir, M. Shah, Image geo-localization based on multiplenearestneighbor feature matching usinggeneralized graphs, IEEE transactions onpattern analysis and machine intelligence 36 (2014) 1546–1558.

[35] T. Weyand, I. Kostrikov, J. Philbin, Planet-photo geolocation with convo-lutional neural networks, in: European Conference on Computer Vision,Springer, 2016, pp. 37–55.

[36] E. Zemene, Y. T. Tesfaye, H. Idrees, A. Prati, M. Pelillo, M. Shah, Large-scale image geo-localization using dominant sets, IEEE transactions onpattern analysis and machine intelligence 41 (2019) 148–161.

[37] G. Capi, M. Kitani, K. Ueki, Guide robot intelligent navigation in urbanenvironments, Advanced Robotics 28 (2014) 1043–1053.

[38] D. Ahmetovic, C. Gleason, C. Ruan, K. Kitani, H. Takagi, C. Asakawa,Navcog: a navigational cognitive assistant for the blind, in: Proceed-ings of the 18th International Conference on Human-Computer Interac-tion with Mobile Devices and Services, ACM, 2016, pp. 90–99.

[39] G. Sargent, K. R. Perez-Daniel, A. Stoian, J. Benois-Pineau, S. Maabout,H. Nicolas, M. N. Miyatake, J. Carrive, A scalable summary generationmethod based on cross-modal consensus clustering and olap cube model-ing, Multimedia Tools and Applications 75 (2016) 9073–9094.

[40] J. R. Quinlan, Induction of decision trees, Machine learning 1 (1986)81–106.

[41] C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20(1995) 273–297.

[42] C. M. Bishop, Pattern recognition and machine learning, Springer Verlag,2006.

[43] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification withdeep convolutional neural networks, in: Advances in neural informationprocessing systems, 2012, pp. 1097–1105.

[44] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally,K. Keutzer, Squeezenet: Alexnet-level accuracy with 50x fewer parame-ters and <0.5 mb model size, arXiv preprint arXiv:1602.07360 (2016).

[45] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).

[46] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: Alarge-scale hierarchical image database, in: Computer Vision and PatternRecognition, 2009. CVPR 2009. IEEE Conference on, Ieee, 2009, pp.248–255.

Egocentric Visitors Localization in Natural Sites · We compare methods based on GPS and vision on the vis-itor localization task in a natural site. Our experiments show that image-based

Documents