Flexible Dialog Management for In-vehicle Dialog Systems

Language Technologies Institute

Flexible Dialog Management

for In-vehicle Dialog Systems

Jeongwoo [email protected]

2


Outline

Introduction Approach in CAMMIA Dialog Management Framework

System Architecture ScenarioTemplate/ScenarioXML ScenarioXML Development Kit

Pilot Systems & Experiments Current Research Focus Future Work

3


Introduction of CAMMIA Project(Conversational Agent for Multilingual Mobile Information Access)

4


Sample DialogS1: How may I help you?U1: I want to go to Carnegie Mellon University.S2: Do you want to go to Carnegie Mellon University?U2: Yes.S3: The distance to the destination is 100 miles. It takes about 2 hours.

U3: I would like to know weather.S4: Please tell me the area and the date.U4: Pittsburgh

(Navigation System sends the next direction to the dialog manager)

S5: To go to Carnegie Mellon University, please make a left turn here. Please tell me the date for Pittsburgh.U5: TomorrowS6: The weather for Pittsburgh tomorrow will be fine.

5


Some Related Research (Speech Interface)

Pellom et al, HLT 2001 (CU-Move) Route planning and navigation Noise suppression front-end and back-end navigation

information retrieval Download driving instructions from the Internet after route

planning User can ask route information during travel (ex. What’s my next turn?)

Coletti et al, IEEE 2003 Hotel retrieval/reservation, POI retrieval, Simple Route

Query Car Wide Web module: XML-based DB interface Local database for tour and geographic data

6


Some Related Research (Multimodal Interface) Minker et al, ICSLP 2002

(SmartKom Mobile) Provide framework for modality control Prototype on Compac iPAG H3630 handheld

computer Apply to pedestrian and driver environment Display maps for route information Slide shows for sight information

7


Approach in CAMMIA Asynchronous communication with navigation

system When getting the next direction, interrupt the current dialog Notify the next turn direction Resume the pending dialog

Maintain dialog history for smooth dialog switching

Flexible & robust Dialog Manager based on VoiceXML (Voice eXtensible Markup Language)

Error handling Correction, Explicit/Implicit confirmation

Support multi-lingual (Japanese, English)

8


Next Introduction Approach in CAMMIA

Dialog Management Framework System Architecture ScenarioTemplate/ScenarioXML ScenarioXML Development Kit


9


ADM

DialogController

Dialog Manager (DM)

Dialog historyDialog scenarioData source

Dialog Management Framework

VXI: VoiceXML InterpreterADM: Asynchronous Dialog Manager

VXI

Julius

Speech Interface

Navigation System

Simulator

Navigation System (NS)

Map Display

VoiceXML

URL to get next direction

HTTP Request

Destination

Direction

Direction

10


Dialog Manager Support multi-user/multi-dialogs Keep track of user session and dialog flows

for smooth task switching Convert user utterances to database query Mixed-initiative interaction Create VoiceXML with dynamic contents => Hard to write VoiceXML by hands => Provide abstract level of dialog

description

11


Dialog Description ScenarioTemplate (ST)

Designed to facilitate new dialog creation Consist of prompts and variables => Dialog designer does not have to know

grammar Support explicit/implicit confirmation Compiled into SXML

ScenarioXML (SXML) Consist of dialog states and transitions Dialog developers need to know grammar, but

don’t have to add other dialogs for dialog switching Compiled into JSP to create VXML with dynamic

contents

12


Example of ST and SXML<variables>weather_area: Stringweather_date: Stringweather_result: String

<states>prompt: Please tell me the area and

date.variable: weather_area weather_date

prompt: Please tell me the area.variable: weather_area

prompt: please tell me the date.variable: weather_date

confirmation: Would you like to know the weather for weather_area weather_date?

response: The weather for weather_date weather_area is weather_result.

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE function SYSTEM "dtd/function.dtd"><function name="Weather"> <state name="ask_weather" position="start"> <grammar src="grammars/ask_weather_jp.gad"/> <prompt> <text>Please tell me the area and date</text> </prompt> <jump> <nextstate next="ask_weather_area"> <field>weather_date</field> </nextstate> <nextstate next="ask_weather_date"> <field>weather_area</field> </nextstate> <nextstate next="confirm_weather"> <field>weather_area</field> <field>weather_date</field> </nextstate> <default>Can you tell me again?</default> </jump> </state> <state name="ask_weather_area" position="transition"> … </state></function>

13


Example of SXML and VoiceXML<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE function SYSTEM "dtd/function.dtd"><function name="Weather"> <state name="ask_weather" position="start"> <grammar src="grammars/ask_weather_jp.gad"/> <prompt> <text>Please tell me the area and date</text> </prompt> <jump> <nextstate next="ask_weather_area"> <field>weather_date</field> </nextstate> <nextstate next="ask_weather_date"> <field>weather_area</field> </nextstate> <nextstate next="confirm_weather"> <field>weather_area</field> <field>weather_date</field> </nextstate> <default>Can you tell me again?</default> </jump> </state> <state name="ask_weather_area" position="transition"> … </state></function>

<?xml version="1.0" encoding="UTF-8" ?><vxml version="2.0" xmlns="http://www.w3.org/2001/vxml"><form><property name="message" value="'SessionID=99 IP=127.0.0.1 PORT=1001 COM=GRAMMAR PARAM=C:\CAMMIA\grammars/;enter_direction_jp.gad;enter_parking_jp.gad;enter_restaurant_jp.gad;cancel_jp.gad;ask_weather_jp.gad;Correction.gad'" /> <block><prompt> Please tell me the area and date</prompt> </block><field name="weather_intention" /> <field name="weather_area" /> <field name="weather_date" /> …<filled namelist="weather_intention"> <if cond="(weather_date!=undefined)"> <submit namelist="recog_result weather_date" next="ask_weather_area.jsp" /> </if> <if cond="(weather_area!=undefined)"> <submit namelist="recog_result weather_area" next="ask_weather_date.jsp" /> </if>…</filled> …</form></vxml>

<field name=“parking_intention" /><field name=“restaurant_intention" /><field name=“direction_intention" />

<filled namelist=“direction_intention"> <goto next="ask_direction.jsp"/></filled>

14


ScenarioTemplate Format

<template> ::= <variables><states><variables> ::= <variable+><variable> ::= <statename>_<id>:<type><statename> ::= <letter+><id> ::= <letter+><type> ::= String|ArrayList<letter> ::= a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|

A|B|C|D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z

<states> ::= <state+><state> ::= {<question+><confirmation><response>}<question> ::= <variable+><prompt><backprompt><prompt> ::= { <letter> | <variable> | " " }<backprompt> ::= <prompt><confirmation> ::= <prompt><response> ::= <prompt>

15


ScenarioXML DTD<?xml version="1.0" encoding="UTF-8"?><!ELEMENT function (state*)><!ATTLIST function name CDATA #REQUIRED><!ELEMENT state (grammar?, result*, prompt?, backprompt?, filled?, jump?, nofound?)><!ATTLIST state name CDATA #REQUIRED position CDATA #IMPLIED><!ELEMENT grammar EMPTY><!ATTLIST grammar src CDATA #REQUIRED><!ELEMENT result (field*, list*, param*)><!ATTLIST result class CDATA #REQUIRED method CDATA #REQUIRED><!ELEMENT field (#PCDATA)><!ELEMENT list (#PCDATA)><!ATTLIST field condition CDATA #IMPLIED><!ELEMENT param (#PCDATA)><!ELEMENT prompt ANY><!ELEMENT text (#PCDATA)><!ELEMENT backprompt ANY><!ELEMENT filled (field*)><!ELEMENT jump (nextstate*, default?)><!ELEMENT nextstate (field*)><!ATTLIST nextstate next CDATA #REQUIRED><!ELEMENT default (#PCDATA)><!ATTLIST default next CDATA #IMPLIED><!ELEMENT nofound (#PCDATA)>

16


SXML Development Kit (SXMLDK) Two-step compilation by SXMLDK

ScenarioXML Development Kit

ST CompilerScenario

Template

SXML Compiler

ScenarioXML

JSP files

Dialog Names Grammars

Voice XML

17


SXMLDK User Interface

18


Next Introduction Approach in CAMMIA Dialog Management Framework



19


Pilot Systems

Prototype I (2002) Focus on building general architecture to support

SXML HTTP is session-less and VoiceXML needs to keep

user’s utterances in each session SXML provides session variable to keep user’s

utterances <session-variables>

<persistent names="DMEnv_persistent_topic DMEnv_persistent_weather_date DMEnv_persistent_weather_area"/>

<transient names="DMEnv_answer_weather_date DMEnv_answer_weather_area DMEnv_answer"/>

</session-variables>

20


Pilot Systems

Prototype II (2003) Improve Prototype I Develop ScenarioTemplate & SXMLDK Filled values and dialog states are stored

in back-end dialog controller Only the current values are stored in

VoiceXML

21


Pilot Experiments Compare Prototype I & II Test environments

VXI: Intel Pentium III 800MHz, 512M RAM DM: Intel Pentium II 400MHz, 224M RAM

Sample Dialog U1: Weather information, please

S1: Please tell me the area and the date U2: Tomorrow S2: Please tell me the area U3: Pittsburgh S3: Are you asking the weather for Pittsburgh tomorrow? U4: Yes. S4: The weather for Pittsburgh tomorrow will be sunny.

date

area

confirmation

response

22


Comparison of Bandwidth Utilization

To compare bandwidth utilization, VXML file sizes were measured

Sub Task Prototype I Prototype II

Ask date 6266 931

Ask area 1949 618

Confirmation 1709 607

Response 3110 283

Avg VXML Size 3258 610

Unit: Byte

23


Comparison of User Wait Time

To compare end-user wait time, loading time in VXI were measured

Sub Task Prototype I Prototype II

Ask date 5.25 0.88

Ask area 3.92 1.28

Confirmation 4.47 1.50

Response 3.44 1.14

Avg Loading Time

4.27 1.20

Unit: second

24


Summary of Prototype II

Improvement Efficient VXML management: VXML includes only

the current state information => faster & less use of network bandwidth Easy to add new dialogs using ScenarioTemplate Easy to support multiple languages by changing

ScenarioTemplate prompts and grammars Issues

Predefined dialog states and prompts Hard to support more complex dialogs

25


Example of Complex DialogU1: Tour guide, please.S1: Tour guidance is started. Where is a destination?U2: Carnegie Mellon UniversityS2: The distance to the destination is 100 miles. It takes about 2

hours.

U3: Do you know a good restaurant near here?S3: What kind of food do you like?U4: I want to eat Italian food.S4: There are Olive garden, Bravo and Laromana.U5: Well, I'd like to go to Laromana.S5: It does not have a parking lot. Is it OK? U6: Tell me the one which has parking lots.S6: Olive garden and Bravo have parking lots. U7: Which one is closer?S7: BravoU8: I would like to go there.S8: Do you want to add it as a way point?U9: YesS9: I set Bravo as the waypoint.

26


Next Introduction Approach in CAMMIA Dialog Management Framework


Pilot Systems & Experiments

Current Research Focus Future Work

27


Current Research Focus

Flexible dialog management Push model: Offer important information even

though user does not request Ex) do not take credit card, do not have parking

lot Search data from the list which was already

retrieved Support comparison (the cheapest, closer) Anaphora resolution (there, it) Add way point to Navigation Map

28


Current Research Focus

Robust dialog management for signal loss Task Manager

- Retrieval task management for signal loss Dialog Manager

- Located in vehicle Info Manager

- Retrieve data from the remote database servers- Maintain local cache with timestamps

29


Extended Architecture

ADM

DialogController

Dialog Manager

Voice

User Interface

InfoManager

Navigation System

Map Display

Julius

Navigation System

Simulator

LocalCache

DB

DataServer

…

VehicleRemote Server

DB

TaskManager

30


Dialog Manager To support information seeking dialogs, it has three

dialog states Search

- Ask the user to fill the minimum search constraints- Display the search resultsEx) There are three Italian restaurants near hear, A, B and C

Search refinement - User can narrow down the searched items with different search

options such as price and distanceEx) Which one is closer?

Selection - Automatically check important information (Push model)- Add way point to the Navigation mapEx) I set it as the waypoint

31


Dynamic State TransitionCheck search

constraints

Display the results

End dialog

need constraints=yes

Ask constraints

need constraints=yes

Selection

Display detailed infoAdd way point if user wants

satisfiedNot satisfied

need features=no

need selection=no

Notify it to the user

Find new lists from the previous resultsDisplay it

Search options

Search options

Check special info

need features=no

Search Refinement

Selection

Search

Search options

Selection

New constraints

32


Future work

More work on Task Manager Anaphora and ellipsis resolution Multi-modal interface

Integration of screen with Navigation map Complementary prompts (voice and screen)

Dynamic grammar generation More intelligent push model based on user

preference Missed turn

Flexible Dialog Management for In-vehicle Dialog Systems

Documents