Feasibility of Interactive Localization and Navigation of People with Visual Impairments Ilias Apostolopoulos, Navid Fallah, Eelke Folmer and Kostas E.

Feasibility of Interactive Localization andNavigation of People with Visual Impairments

Ilias Apostolopoulos, Navid Fallah, Eelke Folmer and Kostas E. BekrisComputer Science and Engineering

1 September 2010, IAS

Wall

Hallway intersection

Door

Motivation

Outdoors• Outdoor navigation systems

typically use GPS for localization

Indoors• Indoor navigation systems

use RFID tags, cameras, laser scanners

• Individuals with VI navigate with compensatory senses (e.g., touch)- Results in reduced mobility

• A need for navigation assistance rises

Easy Challenging

Motivation

• Indoor navigation should be accurate and efficient

• Solution proposed here requires only minimal sensors that can be found in a smart phone.

• Easier to create virtual infrastructure instead of a physical one

Characteristics of the Approach

• User interacts with the phone through an audio/speech interface

• Phone provides directions using landmarks that are recognizable by individuals with VI(i.e. doors, hallway intersections)

• User confirms landmarks through touch

DirectionsAudio Feedback

SpeechLandmark Confirmation

Desired Destination

Challenges and Premises

Challenges• Is there enough

computational power on widely available portable devices?

• Is the sensing information sufficient?

• Can people perform well as sensors in this setup?

Premises• Individuals with VI can recognize

landmarks through touch• Indoor spaces, while complex,

are highly constrained• While pedometers and

compasses are highly erroneous:• They can provide a sufficiently

good estimate for the user’s motion

• When integrated with state of the art methods for localization

Objectives of Feasibility Study

Two research questions

1. Is it possible for a human user to be successfully guided with the overall approach? How do the type of directions provided affect the probability of success?

2. Is it possible to localize the user with an accuracy that will help us adapt directions on the fly?

Background

Localization Techniques

1. Dead-Reckoning• Integrate measurements of the human motion• Accelerometers and radars have been used

- P. Van Dorp et al., Human walking estimation with radar, 2003

• Error grows unbounded

2. Beacon-based• Use identifiers in the physical space• Beacons can be detected by cameras, infrared or ultrasound

identifiers• Popular solutions use RFID tags

- V. Kulyukin et al., Rfid in robot-assisted indoor navigation for the visually impaired, 2004

• Locating beacons might be hard and inaccurate• Significant time and cost spent installing and calibrating beacons

Localization Techniques

3. Sensor-based• Use cameras to detect pre-existing features (e.g., walls,

doors)- Cameras need good lighting and have prohibitive computational cost- O. Koch and S. Teller, Wide-area egomotion estimation from known

3d structure, 2007

• Use 2D laser scanners- Expensive and heavy- J. A. Hesch et al., An indoor localization aid for the visually impaired,

2007

• The system proposed is a sensor-based system• Pedometer and compass for localization• User as a sensor for landmark confirmation• Use of probabilistic tools from robotics

Bayesian Methods from Robotics

• Candidate methods:1. Extended Kalman Filter

• Assumes normal distribution+ Returns optimum estimate under certain assumptions- Not a good model for multimodal distribution• R. E. Kalman, A new approach to linear filtering and prediction

problems, 1960

2. Particle Filter• Employs a population of discrete estimates+ Can represent a multimodal distribution- High computational cost• N. J. Gordon et al., “Novel approach to nonlinear/non-gaussian

bayesian state estimation”, 1993

• Particle Filter is chosen due to better accuracy about the location of the user

Methodology

Part 1: Directions

High-level Operation

1. A user specifies a start and destination room number to travel to.

2. The system computes the shortest path using A* and finds landmarks along the path.

3. Directions are provided iteratively upon completion through the phone’s built-in speaker. The user presses a button on the phone after successfully executing each direction.

Types of Directions

• Metric based

Intersection

Door Door15 steps

20 steps

• Landmark based

Directions

Metric based Landmark based

No Max Threshold

9 Meters Threshold

15 Meters Threshold

No Max Threshold

9 Meters Threshold

15 Meters Threshold

Direction Provision

Landmark No Max Threshold"Exit the room then turn right""Move forward until you reach a hallway

on your left“"Turn left to the hallway“"Move forward until you reach a water

cooler on your left“"Move forward until you reach a hallway

on your left“"Turn left to the hallway""Follow the wall on your left until you

reach the third door“"You have reached your destination"

Direction Provision

• Landmark 15 Meters Threshold• "Exit the room then turn right“• "Move forward until you reach a hallway

on your left“• "Turn left to the hallway“• "Follow the wall on your left until

you reach the third door“• "Move forward until you reach a water

cooler on your left"• "Move forward until you reach a hallway

on your left"• "Turn left to the hallway"• "Follow the wall on your right until you

reach the 5th door"• "Follow the wall on your left until you

reach the first door"• "You have reached your destination"

1

2

3

1 2 3 4 5

Direction Provision

• Landmark 9 Meters Threshold• "Exit the room then turn right"• "Follow the wall on your right until you reach

the first door"• "Move forward until you reach a hallway on

your left“• "Turn left to the hallway"• "Follow the wall on your left until you reach

the second door“• "Move forward until you reach a water cooler

on your left“• "Move forward until you reach a hallway on

your left"• "Turn left to the hallway"• "Follow the wall on your right until you reach

the third door“• "Follow the wall on your left until you reach

the first door"• "You have reached your destination"

Direction Provision

• Metric No Max Threshold• "Exit the room then turn right"• "Walk x steps until you reach a

hallway on your left“• "Turn left to the hallway“• "Walk x steps until you reach a water

cooler on your left“• "Walk x steps until you reach a

hallway on your left“• "Turn left to the hallway“• "Walk x steps until you reach a door

on your left“• "You have reached your destination"

Direction Provision

• Metric 15 Meters Threshold• "Exit the room then turn right“• "Walk x steps until you reach a hallway on

your left"• "Turn left to the hallway“• "Walk x steps until you reach a door on

your left“• "Walk x steps until you reach a water

cooler on your left“• "Walk x steps until you reach a hallway on

your left“• "Turn left to the hallway“• "Walk x steps until you reach a door on

your right“• "Walk x steps until you reach a door on

your left"• "You have reached your destination"

Direction Provision

• Metric 9 Meters Threshold• "Exit the room then turn right";• "Walk x steps until you reach a door on your

right“• "Walk x steps until you reach a hallway on your

left“• "Turn left to the hallway“• "Walk x steps until you reach a door on your

left“• "Walk x steps until you reach a water cooler on

your left“• "Walk x steps until you reach a hallway on your

left“• "Turn left to the hallway";• "Walk x steps until you reach a door on your

right“• "Walk x steps until you reach a door on your

left“• "You have reached your destination"

Methodology

Part 2: Localization

Localization

• Objective is an accurate location of the user• Previous location and sensor data are used as input• The new location is the output of the system

• Model• n static landmarks• ξ = (x, y, θ) state of the system• m map that stores information about the world• Landmarks belong to k different types (e.g, doors,

hallway intersections) Landmarks in the same class are indistinguishable

• Data dT = (o(0:T), u(0:T-1)) u(0:T-1) transitions o(0:T) observations

Transitions and Observations

Transitions• A transition

corresponds to a motion where the agent acquires the orientation and moves forward

Observations• Observation of a

landmark from state

implies:•

(xt+1,yt+1,θt+1)

utf ut

θ

(xt,yt,θt)

(xt,yt)

Robs

li

(xi ,yi)

Filtering

• The objective is to be able to incrementally estimate the user’s state at time T.

• Given the normalization factor η the belief distribution can be updated as follows:•

PreviousBelief

Transition modelObservation model

x

x+dxTransition

x'

Applying Transition

model

filterwith

observation

Particle Filter

• It is possible to represent BT through a set P of particles

• Each particle stores a state estimate together with a weight

• At each time step T, the following steps are:A. For each particle

i. Employ the transition model to acquire

ii. Employ the observation model to compute the new weight

B. Sample a new population of P particles given the weights

Implementation of Transition Model

• Collects orientation from compass and step counts from pedometer• Multiple readings from compass are averaged for a time

step• Pedometer returns 1 if the user moved a step, else 0

• Step length is calculated for each user through some train paths

• Noise is added with a normal distribution to introduce uncertainty

Implementation of Observation Model

• Two cases:• No landmark detected by user

• All particles get weight 1• Landmark detected

• Particles within observation range get weight inversely proportional to distance

• Particles out of range get weight 0

Sampling

• Algorithm samples with higher probability particles with higher weight

• When all particles get a weight of 0:• Algorithm finds the closest visible landmark of the type

observed by the user for each particle• Particle is sampled near the landmark

Experiments

Setup

• System was implemented in Java for Google’s Android platform

• Map of a buildings floor in campus was created in Keyhole Markup Language(KML)

• Map contains:• 3 water coolers• 1 floor transition marked by a metal strip• 3 hallway intersections & 2 hallway turns• 72 doors

• 5 paths were defined• two alternatives for directions with three granularities

for each

Participants

• 10 volunteers• Users held the phone in one hand and a cane in the

other• 1 volunteer was legally blind and assisted with the

setup of the experiments• 9 more sighted volunteers were blindfolded• Some users had prior knowledge of the building

while others didn’t• Each user executed ten traversals• Two traversals per path using different directions

Ground Truth

• An observer was recording the user’s motion• Markers were placed on the floor every two meters• Every time the user was crossing a marker the

observer was recording the time in a second smart phone

• Assume that user moves with constant speed between markers

• Ground truth resolution• 2 meters

Objectives of Feasibility Study

Two research questions

1. Is it possible for a human user to be successfully guided with the overall approach? How do the type of directions provided effect the probability of success?

2. Is it possible to localize the user with an accuracy that will help us adapt directions on the fly?

1. Success Ratio of Direction Provision

• 84% of experiments reached a distance smaller than 2 meters

• 92% of experiments reached less than 3.5 meters

Distance from goal

Path 1 Path 2 Path 3 Path 4 Path 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Landmark No Max Threshold Landmark 9 Meters ThresholdLandmark 15 Meters Threshold Metric No Max ThresholdMetric 9 Meters Threshold Metric 15 Meters Threshold

meters

1. Success Ratio of Direction Provision

• Duration of execution

Path1 Path 2 Path 3 Path 4 Path 50

50

100

150

200

250

300

Landmark No Max Threshold Landmark 9 Meters Threshold Landmark 15 Meters ThresholdMetric No Max Threshold Metric 9 Meters Threshold Metric 15 Meters Threshold

sec

2. Localization Accuracy

• Particle filter improves considerably results acquired just by sensor readings

• Paths with distinctive landmarks had considerably lower error than paths with repetitive landmarks

Distance from Ground Truth

Localization Error

Average error: 3.69 meters

2. Localization Accuracy

Path 1 Path2 Path 3 Path 4 Path 50

5

10

15

20

25

30

Landmark No Max Threshold Landmark 9 Meters ThresholdLandmark 15 Meters Threshold Metric No Max ThresholdMetric 9 Meters Threshold Metric 15 Meteres Threshold

met

ers

Discussion

• Proposed a minimalistic indoor navigation system for individuals with VI

• Sensors used are inexpensive and common though erroneous

• Answers to research questions:• It is possible for an individual with VI to be guided

successfully with this scheme• It is possible to localize the user with accuracy

Update

Submitted version Current version

Path 1 Path2 Path 3 Path 4 Path 50

5

10

15

20

25

30

Landmark No Max Threshold Landmark 9 Meters Threshold

Landmark 15 Meters Threshold Metric No Max Threshold

Metric 9 Meters Threshold Metric 15 Meteres Threshold

me-

ters

Path 1 Path 2 Path 3 Path 4 Path 50

5

10

15

20

25

30

Landmark No Max Threshold Landmark 9 Meters ThresholdLandmark 15 Meters Threshold Metric No Max ThresholdMetric 9 Meters Threshold Metric 15 Meters Threshold

me-

ters

Average error: 3.69 meters Average error: 2.05 meters

Future Work

• Automatic direction provision based on localization estimates

• User studies with a larger number of users with VI• More complex environments• Buildings with multiple floors• Elevators, stairs, ramps

• 3D maps with more details about the world

Thank you for your attention!