H OW M ULTI - ROBOT F ORAGING S CALES WITH N UMBER OF R OBOTS Prasanna Velagapudi Paul Scerri, Katia Sycara Robotics Institute, Carnegie Mellon University.

Speaking Qualifier - Nov. 24, 2009

HOW MULTI-ROBOT FORAGING SCALES WITH NUMBER OF

ROBOTS

Prasanna VelagapudiPaul Scerri, Katia Sycara

Robotics Institute, Carnegie Mellon University

Huadong Wang, Michael Lewis

Dept. of Information Sciences, University of Pittsburgh

Speaking Qualifier - Nov. 24, 2009 2

Outline

• Motivation• Problem• Related Work• Experiment• Results• Discussion/Conclusion


Motivation

Perimeter PatrolsDisaster response Search and Rescue


Motivation

• Goal: Increase operator’s span-of-control– Span-of-control: the number of subordinates (robots) a

supervisor (operator) has

• Human operators are necessary– Complex perception, meta-knowledge– Supply high-level, dynamic goals

• Humans get overloaded


Problem: Improving Span-of-Control

• How can we increase human operators’ span-of-control?

– How does task performance vary with number of robots?

– Which tasks are most limiting to the operator?

– Does alleviating a task improve performance?


Understanding the Task


Decomposing Foraging

Exploration Perceptual Search

Foraging is composed of two largely independent but concurrent subtasks:


Focusing the Problem

• Multi-robot foraging with waypoint control– Widely cited as likely field application– Each robot searches its own region

• Minimal coordination to avoid overlaps or gaps– Waypoint control:

• Lowest level of automation compatible with independent control of multiple robots


Related Work

Study Task World Robots InteractionTrouvain & Wolf (2002):User study of the impact ofrobot group size

Navigation 2D simulatedoffice world

2, 4, 8 UGVs(homogeneous)

Waypoint

Trouvain et al. (2003): User study of map based and camera based user interface

Exploration 3D simulatedoutdoor world

1, 2, 4 UGVs(homogeneous)

Supervisory + waypoint control

Olsen & Wood (2004): Fan-out independent study

Foraging 2D simulatedoffice like world

18 UGVs (homogeneous)

Waypoint

Humphrey et al. (2007):User study of robot team & halo interface

Foraging 3D simulatedoutdoor world(USARSim)

6, 9 UGVs(heterogeneous)

Teleoperation and scripted behaviors

Nehme et al. (2008): Impact of Heterogeneity on Operator Performance

Foraging 2D simulated 6, 8,12 UGVs(heterogeneous)

WaypointAutomation


Related Work: Span-of-control Limits

• For foraging, operator span-of-control limits fall between 4-9 robots – Olsen & Wood (2004), Humphrey et al. (2007)

• Dependent on environmental demands• Generally, operators can use more robots if robots

have a higher level of autonomy


Related Work: The Fan-out Plateau

If a robot is added to a team, the change in performance is proportional to robot’s independence and operator’s available cognitive resources.

– Olsen & Wood (2004)Pe

rfor

man

ce

# of Robots

Performance plateau when operator saturates

Diminishing returns as # of robots increases


Fan-out Hypothesis

• Task performance should follow Fan-out model

Perf

orm

ance

# of Robots

Performance Plateau

Diminishing returns


Subtask Hypothesis

• Task performance will reflect which subtask is contributing more to operator workload

Perf

orm

ance

# of Robots

Subtask 1

Subtask 2 (limiting)

Full task

Exploration

Perceptual Search


Experiment

• Task: Simulated search and rescue• Single operator controlling 4, 8, and 12 robots

– Waypoint control (primary)– Direct teleoperation

• Perform either full foraging task or just one subtask (exploration or perceptual search)


USARSim

[http://www.sourceforge.net/projects/usarsim]

• NIST-maintained open source simulator

• High-fidelity physics• “Realistic” rendering

– Camera– Laser scanner (LIDAR)– IMU/Odometry


Experiment MapP2AT Robots

Human “Victims”(24 total)


MrCSMulti-robot Control System



Waypoint Navigation

Teleoperation

Video/ Image Viewer

Status Window

Map Overview




Conditions

Exploration• Issue waypoints

Perceptual Search• Locate victims

Full Task• Issue waypoints• Locate victims


Experimental Procedure

• Between groups, repeated measure design

Full Task Exploration Perceptual Search

DemographicsTraining 20min4 robots 15min

Workload Survey

8 robots 15min Workload Survey

12 robots 15minWorkload Survey

Each

Sub

ject


Methodology

• Independent Variables:– Conditions of Task– Numbers of Robots

• Dependent Variables:– Workload (NASA-TLX)– Victims found– Area Explored– Switches in focus among

robots– Number of assigned

missions– Average path length– Robots neglected


Participants

• 44 paid participants from U. of Pitt. community– 25 male, 20 female– Ages 20-33 (average age 25.18)

• No prior experience with robot control• Most were frequent computer users


Conditions

Exploration• Issue waypoints

Perceptual Search• Locate victims

Full Task• Issue waypoints• Locate victims


Task Performance: Full Task vs. Perceptual Search

• Compare number of victims found in full task and perceptual search conditions

• Expected: Fan-out plateau and performance gapPe

rfor

man

ce

# of Robots

Perceptual search

Full task


Task Performance: Full Task vs. Perceptual Search

F1,28 = 27.4

p < .0001

Diminishing return/plateau

Operator overload


Task Performance:Full Task vs. Exploration

• Compare total area explored in full task and exploration condition

• Expected: Fan-out plateau and performance gapPe

rfor

man

ce

# of Robots

Exploration

Full task


Task Performance:Full Task vs. Exploration

F1,28 = 21.17

p < .002

Diminishing return/plateau

Operator overload

Similar performance


Fewer pathsLonger paths

4 Robots 8 Robots 12 Robots20

25

30

35

40

45

fulltask

Exploration Control Statistics

# Paths Issued Average Path Length

# of

Pat

hs

Leng

th o

f Pat

h (m

)

4

4.5

5

5.5

6

6.5

7

7.5

8

8.5

fulltask

4 robots 8 robots 12 robots



0.20.40.60.8

11.21.41.61.8

2

Neglected fulltaskNeglected exploration

Neglected Robots

• Neglected robots are never issued waypoints• Further evidence of reaching operator span-of-control limits

χ22 = 10.75

p = 0.005

# N

egle

cted

Rob

ots


Number of Switches Between Robots

• Before operators issue waypoints or mark victims, they must switch to the desired robot

• Switches are cognitively expensive to operatorsExploration Perceptual Search


Close correspondence

Number of Switches Between Robots

F2,54 = 12.6 p < .0001

Num

ber o

f Sw

itche

s Be

twee

n Ro

bots

Significant gap

10

15

20

25

30

35

40

45

50

55

perceptual search explorationfulltask

4 robots 8 robots 12 robots


Operator Pause Statistics

• Robots become paused when commanded by operator or when finished with all waypoints– Perception pauses

• Operator stops robot in middle of mission to locate victim– Navigation pauses:

• Robot is paused by itself or by operator so that a new set of waypoints can be issued

• Pauses may indicate operator neglect


Distribution of Pauses

Full Task Exploration Perceptual Search

4 8 12 44 8 812 12

# of Robots

Pause (sec)

More frequent pauses

More long-duration pauses

(robot neglect)

Coun

ts


Aggregate Pause DurationAg

greg

ate

Paus

e D

urati

on p

er R

obot

(s

ec)


Workload Survey

• Survey (NASA-TLX) administered after each session (4, 8, 12 robots)

• Aggregate score indicates participants’ subjective assessment of cognitive workload

• Expected:

Wor

kloa

d# of Robots

Subtask 1

Subtask 2 (limiting)

Full task


Workload Survey

F1,27 = 21.17p < .0001

Significant gap


Discussion

Fan-out Hypothesis: Task performance should follow Fan-out model

• Performance in subtasks consistent with fan-out

• Performance in full task does not follow fan-out– Inflection point in performance at 8 robots– Operators experiencing cognitive overload


Discussion

Subtask Hypothesis:Task performance will reflect which subtask is contributing more to operator workload

• Exploration and full task performance similar– Removing perceptual search subtask removes little

workload• Perceptual search performs better than full task

– Significantly less workload than other tasks with 12 robots


Conclusions

– How does task performance vary with number of robots?Fulltask: operators overload, performance dropsSubtasks: follows fan-out model

– Which tasks are most limiting to the operator?The exploration subtask

– Does alleviating a task improve performance?Removing exploration helps, removing perceptual search does not


Conclusions

• How can we increase human operators’ span-of-control?

• Offload (automate) Exploration subtask– When operators are overloaded with full task, performance

drops dramatically– Operators devote most cognitive effort to exploration,

performance not much better than full task


Future Work

• Effects of exploration autonomy– How is span-of-control affected by higher-level autonomy,

sliding autonomy?• Effects of perception/exploration task difficulty

– Do scaling effects hold in more realistic environments?• Scaling to larger team sizes (16, 24+)

– Do subtasks also have steep performance drop-offs?• Multi-operator interaction

– Can subtasks be distributed across operators?


Questions?



Related Work - Supplemental


Methodology – Perceptual search

• If an autonomous path planner is used:– Covers a larger area than possible with a human operator (never pauses upon

arrival at a waypoint)

• If human generated trajectories are taken from the full task:– Pauses for waypoint completion– Pauses at locations where victims were found

• Solution: Use trajectories from the exploration condition– Contain pauses associated with waypoint arrival– Do not contain pauses for identifying victims.– Operators must be able to pause robots when they discover victims

Learning curves

20 35 20 35 50

6520 35 50

4 Robots 8 Robots

12 Robots

100%

100%

100%



25

30

35

40

45

fulltask exploration

Missions Assigned

F1,28 = 6.34

p < .018

Mis

sion

s as

sign

ed

Operator overload


Average Path Length


4.5

5

5.5

6

6.5

7

7.5

8

8.5

fulltask exploration

Aver

age

Path

Len

gth

(m)


USARSim Validation Studies

• Synthetic video– Carpin, S., Stoyanov, T., Nevatia, Y., Lewis, M. and Wang, J. (2006a). Quantitative assessments of

USARSim accuracy". Proceedings of PerMIS 2006• Hokuyo laser range finder

– Carpin, S., Wang, J., Lewis, M., Birk, A., and Jacoff, A. (2005). High fidelity tools for rescue robotics: Results and perspectives, Robocup 2005 Symposium.

• Platform physics & behavior– Carpin, S., Lewis, M., Wang, J., Balakirsky, S. and Scrapper, C. (2006b). Bridging the gap between

simulation and reality in urban search and rescue. Robocup 2006: Robot Soccer World Cup X, Springer, Lecture Notes in Artificial Intelligence

– Lewis, M., Hughes, S., Wang, J., Koes, M. and Carpin, S., Validating USARsim for use in HRI research, Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society, Orlando, FL, 457-461, 2005.

– Pepper, C., Balakirsky, S. and Scrapper, C., Robot Simulation Physics Validation, Proceedings of PerMIS’07, 2007.

– Taylor, B., Balakirsky, S., Messina, E. and Quinn, R., Design and Validation of a Whegs Robot in USARSim, Proceedings of PerMIS’07.

– Zaratti, M., Fratarcangeli, M., and Iocchi, L., A 3D Simulator of Multiple Legged Robots based on USARSim. Robocup 2006: Robot Soccer World Cup X, Springer, LNAI, 2006.

[http://www.sourceforge.net/projects/usarsim]


Olsen & Wood 2004 Interface


Nehme et al. 2008 Interface


NASA-TLX Workload Survey


Conclusions

• Operators devote most cognitive effort to exploration– Automating exploration is necessary to increase operator

span-of-control when foraging– When operators are overloaded, performance can drop

dramatically

• Operators may be able to handle more than 12 robots when doing perceptual search alone– Most appropriate role for operator is sensor and

information fuser– Need control approaches that are observation, rather than

navigation, oriented


Conclusions

• According to H1, full task performance drops more than expected as team size increases

• According to H2, exploration is largest contributor to operator workload

• Operator span-of-control limit between 8, 12• For larger teams, automate exploration

H OW M ULTI - ROBOT F ORAGING S CALES WITH N UMBER OF R OBOTS Prasanna Velagapudi Paul Scerri, Katia Sycara Robotics Institute, Carnegie Mellon University.

Documents

control span

overloaded speaking

control limits

gaps waypoint control

number of robots

robots performance pla

operator span

robots olsen wood