Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems

Uncertainty Corpus: Resource to Study User Affect in Complex

Spoken Dialogue Systems

Kate Forbes-Riley, Diane Litman, Scott Silliman, Amruta Purandare

University of PittsburghPittsburgh, PA, USA

2

Outline

Introduction

WOZ-TUT System

Experimental Design

Uncertainty Corpus Description

Uses of the Uncertainty Corpus

3

Overview: Towards Affect-Adaptive Spoken Dialogue Systems

Automatic Detection: promising across affective states and applications, e.g. (Craig et al., 2006; Litman & Forbes-Riley, 2006; Lee & Narayanan, 2005; Vidrascu & Devillers, 2005; Batliner et al., 2003)

Larger goal is automatic adaptation, but results are sparser

More public affect-annotated corpora of human-computer dialogues could help, e.g. HUMAINE project

SYMPAFLY, AIBO (Batliner et al., 2004) (German) Communicator (Walker et al., 2001; Ang et al., 2002) (English)

Uncertainty Corpus: new complex domain: spoken dialogue tutoring new affect annotation: student uncertainty

4

Uncertainty Corpus Collection: WOZ-TUT System

WOZ-TUT: Adaptive Wizard of OZ Tutoring System modified version of ITSPOKE (Litman & Silliman, 2004)

system responses based on combined uncertainty and correctness human recognizes speech, annotates uncertainty and correctness

Why uncertainty?

Most frequent in ITSPOKE corpora (Forbes-Riley & Litman, 2007)

Most systems respond only to correctness, but literature suggests uncertain and incorrect answers signal learning impasses

What uncertainty adaptation?

Treating uncertain+correct answers as incorrect should provide additional knowledge to bridge impasse

5

WOZ-TUT Screenshot

6

Experimental Design

3 Conditions: used parameterized WOZ-TUT dialogue manager

Experimental: treat all uncertain+correct turns as incorrect

First Control: ignore uncertainty (logged)

Second Control: ignore uncertainty (logged), but treated a percentage of random correct answers as incorrect

7

TUTOR: What will the velocity of the object be a second after that (where the initial velocity is 9.8m/s and the acceleration is 9.8m/s2)?

STUDENT: Nineteen point six meters per second?? [uncertain+correct]

TUTOR in First Control Condition moves on: Good. So at every point in time during the fall of the man and his keys, how do their velocities compare with each other?

TUTOR in Experimental Condition remediates: Okay. As we have seen, if a falling object has an acceleration of 9.8m/s2, its velocity changes by 9.8m/s every second. So if a second after it began falling its velocity is 9.8m/s, a second later its velocity will be 9.8m/s + 9.8m/s = 19.6m/s. So what will its velocity be a second after that?

Corpus Excerpts

8

Experimental Procedure

60 subjects randomly assigned to 3 conditions (gender-balanced)

Native English speakers with no college physics

Procedure: 1) read background material, 2) took pretest, 3) worked training problem with WOZ-TUT, 4) took posttest, 5) worked isomorphic test problem with non-adaptive WOZ-TUT

9

Corpus Description

120 dialogues from 60 students (.ogg format) 20 total hours of dialogue Student turns manually transcribed, including disfluency and

non-syntactic question annotation Tutor turns and Wizard annotations in log files

Student Tutor

Total Turns 2171 2531

Total Uncertain Turns 796 -

Total Words 13533 111829

Average Words per Turn 6.23 44.20

10

Student Answer Attributes

One-way ANOVAs showed no significant differences: number of correct, uncertain, or uncertain+correct turns number adapted-to turns (EXP vs CTRL2)

Training Problem EXP CTRL1 CTRL2

Ave Turns 20.65 18.60 19.75

Ave Correct Turns 13.80 12.55 14.20

Ave Uncertain Turns 9.95 8.60 11.15

Ave Uncertain+Correct Turns 4.75 3.75 6.10

Ave Adapted-To Turns 4.75 0 3.65

Ave Uncertain+Correct and Adapted-To Turns

100% 0% 36%

11

Uses of the Uncertainty Corpus I

Isomorphic Test Problem EXP CTRL1 CTRL2

Ave Turns 16.50 16.80 16.25

Ave Correct Turns 14.60 14.35 14.10

Ave Uncertain Turns 3.30 3.15 3.65

Compare student performance across conditions to isolate impact of uncertainty adaptation No significant differences in learning We are comparing dialogue-based metrics in the isomorphic test problem (Forbes-Riley, Litman and Rotaru,

2008)

- Feedback confound identified and rectified in larger follow-on study

12

Uses of the Uncertainty Corpus II Resource for analyzing linguistic features of naturally-

occurring user affect in human-computer dialogue Models built from elicited emotions generally transfer poorly to

naturally-occurring dialogue (Cowie and Cornelius, 2003; Batliner et al., 2003)

Uncertainty Corpus provides a new resource for modeling naturally-occurring dialogue

Large number of features in speech, transcript, log files

13

Summary and Current Directions The Uncertainty Corpus is a collection of tutorial

dialogues between students and an adaptive Wizard-of-Oz spoken dialogue system

Corpus (speech, transcripts, uncertainty and correctness annotations) publicly available by request through the Pittsburgh Science of Learning Center: https://learnlab.web.cmu.edu/datashop/index.jsp

Follow-on experiments and corpora Larger WOZ study just completed, with learning results! Fully automated study to begin Fall 2008

14

Questions?

Further Information?

web search: ITSPOKE or PSLC

Thank You!