Heterogeneous Multi-Robot Dialogues for Search Tasks Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky AAAI Spring Symposium 2005: Dialogical.

Heterogeneous Multi-Robot Dialogues for Search Tasks

Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky

AAAI Spring Symposium 2005: Dialogical Robots

AAAI Spring ’05 Symposium: Dialogical Robots 2

Communication among autonomous robots and humans

→ Embodied (not necessarily robotic, or even

anthropomorphic) agents will become ubiquitous

→ NL dialogue will be a useful modality

→ Agent’s dialogue and task models will remain tightly coupled, and independent from other agents

We will be talking to multiple agents without a (completely) shared dialog system


Gedankenexperiment

• The problem of many robots: sweeper, furniture mover, baby monitor.

• Task: clean up house; don’t wake baby up.

• Who do you talk to?• What do you say?• Who talks back?


Issues for polylogical systems

• What’s a reasonable architecture?– A single system controlling multiple robots– A single DM directing autonomous robots– Shared understanding/generation– Shared microphone and speakers– Robots self-contained: Nothing shared

• How to communicate with multiple agents?– Directions to the entire team? (“talk into the air”)– Mediation through a designated team leader? (foreman)– Instructions to each team member? (direct management)– To a proxy agent within/without the team? (personal assistant)


What do we have today?

• A platform that supports multiple (robot) participants– Sphinx II ASR; Festival TTS– RavenClaw dialogue manager– Galaxy-II message-passing architecture– Several “back-end” interfaces

• A basic corridor movement domain– Commands to move robots along in vectors and

towards named locations– Mechanisms for describing robot status and location

to human interlocutor


AGENT2

AGENT1

HUMAN INTERFACE

Current Architecture

ASR

Multi-voice TTS

DM1

DM2

Robot1

Robot2

Interpret1

Interpret2


AGENT2

AGENT1HUMAN INTERFACE

Implementation

Sphinx II Phoenix Helios

Galaxy

RavenclawBashful

RavenclawClyde

Clyde

Bashful

Robot Communication System

RosettaSwift


Human-Robot Treasure Hunt

• One Human

• Two robots– Segway– Pioneer

• Treasures “hidden” in large space

• TASK: Retrieve all treasures


Scenario for Search

We found it!

We are at <x,y>


Issues in H-R communication

1. How do people decompose the task into sub-tasks?

2. What language do people use to get the tasks performed by the robots?

3. Given a human command, what is the expected robot behavior?

Explore using Wizard of Oz experiments


WOZ design

Communication takes place by normal speech, walkie-talkie, and through the webcam.

•1 experimenter, 2 robot actors, 1 participant.•Experimenter places treasure, simulates robot treasure sensors.•1st robot actor is blind, but carries a webcam for the participant’s consumptions.•2nd robot actor can only move by crawling.


Annotation and analysis

Data transcribed and annotated

Utterances classified into functional categories

394 utterances

20 utterance categories

8 major categories

Carnegie Mellon MockBrow annotation tool


Utterance/Task Breakdown

Controlling team behaviors Grounding

Positive/negative feedback Informing robot of it’s state or

the world Explanations of commands Orientation Grounding

Navigation Simple Navigation commands Spatial Referential Navigation Object Referential Navigation

Manipulation Manipulating the environment Manipulating treasure

Coverage Manipulating the webcam view Object coverage commands Generic coverage

Asking about the robot’s abilities Filler Real-time command

modifications


Summary / Future Work

• Architecture for human-robot teams

• WoZ study of language requirements

• Different WoZ scenarios

• Implement language for robot team

• “Field” testing


Controlling team behaviors

“you guys get together” “T- you go first and B- follow”


Grounding

Positive/negative feedback “ok that’s better”

Informing robot of state “so that’s up” “I don’t see anything there”

Explanations of commands “so I can see which direction is up”

Orientation Grounding “What you’re facing now with the camera – is that the

vehicle that you just circumnavigated” “I can tell you’re going in the wrong direction, stop”


Navigation

Simple Navigation commands “so um T- turn to you left” “T- I want you to turn right 90 degrees” “can you go in that general direction” “can you proceed in that direction”

Spatial Referential Navigation “go to that open area” “continue around the periphery of that open area” “back out of that alley” “proceed in that direction until you find an opening to turn left”

Object Referential Navigation “go over by T-” “can you go on the other side of that vehicle” “go over by the posters”


Manipulation

Manipulating the environment “T- why don’t you move the trash can”

Manipulating treasure “T- bring the coin to me”

Manipulating the webcam view “ok B- look to your left” “B- can you look around with the camera a

little”


Coverage

Object coverage commands “ok so examine the shelf” “do you see something on that shelf in front of

B-” “can you look over by that table over there”

Generic Coverage “do you see anything that looks interesting”


Asking about the robot’s abilities

“is that possible”


Filler

“and now um” “ok um”


Real-time command modifications

“keep going” “stop” “a little more” “change of plans” “other direction”

Heterogeneous Multi-Robot Dialogues for Search Tasks Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky AAAI Spring Symposium 2005: Dialogical.

Documents

dialogical robots5

multiple robots

human interlocutor slide

autonomous robots

robot status

robot actors

dialogical robots4 issues

dialogical robots10