Cooking an ontology-based spoken dialogue system

Cooking an ontology-based coaching dialogue system

— based on my PhD proposal —

Joana Paulo PardalQuality and Usability ColloquiumDecember 7th, 2009Berlin, Germany

Monday, December 7, 2009

Who am I?Monday, December 7, 2009

Researcher, PhD Student and LecturerRESEARCHER:

- Spoken Language Systems Laboratory (L2F) @ INESC-ID, Lisbon

PHD STUDENT: - IST, Technical University of Lisbon- Ontology-based (spoken) Dialogue Systems

LECTURER:- IST, Technical University of Lisbon



cooking a coaching

ontology-based (spoken) dialogue systems


Cooking


House of the future



Part of our daily life


With a lot of inspiration


And interested people


Eyes and hands busy


Dialogue Systems Monday, December 7, 2009

Typical Architecture

back-endsystem

DialogueManagement

Speech Recognition

Natural Language

Understanding

Natural Language

Generation

Speech Synthesis

User


Typical Architecture

acoustic models

grammar

back-endsystem

DialogueManagement

Speech Recognition

Natural Language

Understanding

Natural Language

Generation

Speech Synthesis

User

lexicon

grammar

lexicon

grammar

prosody

coded speech


IO Manager Dialogue Manager

Se

rvic

e M

an

ag

er

JV

oic

eX

ML &

JS

pe

ech

AP

I 2

.0 in

terf

ace Parser

Interpretation

Manager

Behavioral

Agent

Generation

Manager

Surface

Generation

Discourse Context

Ta

sk

Ma

na

ge

r

Dia

log

ue

Ma

na

ge

r In

terf

ace

De

vice

Co

re

Access

Manager

Device

Manager

Core

Business

De

vice

Co

reD

evi

ce

Co

re

QA@L2F

Data

base

User

ASR(Audimus)

GUI

TTS(DIXI)

FaceEvent Manager

Microphone

Array

DigA L2F INESC-ID


TRIPS Univ Rochester


!"

#$%&'()*+,-&./0$12%02,/%

3405%67

!"#$%&'(&

)#*+,&

-.#"&/0

1&234#5

!"#$%&%$#'#()"!"#$%&'

*('+),-.'"',$%()*$%+,)-

/$"$%'#()"(#.$//)

0$1),"(#()")01&#.$(*$(

23"#4$5(52),,&#!$

!"#$%'1#()"-.,%)!#,,#

8)62$9&:6;16%!"#$%&

<%0=;9&:6;16%!"$'%()&

Olympus Carnegie Mellon


Practicaldialogue


http://flickr.com/photos/carpathian/1656738696/in/photostream/

Practical Dialogue HypothesisThe conversational competence required for practical dialogues, while still complex, is significantly simpler to achieve than general human conversational competence.


Domain-Independence HypothesisWithin the genre of practical dialogue, the bulk of the complexity in the language interpretation and dialogue management is independent of the task being performed.


Types of dialogue systems:- Information providing- Appliances control- Planning & scheduling- Tutoring...


Coaching

Tehran Times Sports Desk


Simple but funcional dialogue systems


How different levels of knowledge stored in domain-specific ontologies can be used to facilitate the creation

of new coaching dialogue systems capable of domain reasoning?

http://www.flickr.com/photos/jhhwild/1781000505/Monday, December 7, 2009

http://flickr.com/photos/richard_am/


short introduction to

ONTOLOGIES


on·tol·o·gy

ORIGIN early 18th cent.:from modern Latin ontologia, from Greek ONT- ‘being’ -LOGY.

(philosophy) the branch of metaphysics that addresses the nature or essential characteristics of being and of things that exist; the study of being in the capacity of being.

(computer science) a structure of concepts or entities within a domain, organized by relationships; a system model.

Definition adapted from http://en.wiktionary.org/wiki/ontology


is a description of the existing concepts and relationships in a given domain

“An ontology is a specification of a conceptualization.”

[Gruber, 1993]

What is an Ontology?



Tim Berners-Lee’s Semantic WebMachine processable (not natural language), for data, for the future.

http://www.w3.org/2000/Talks/1206-xml2k-tbl/slide10-0.html


machine processable and

interoperable systems

http://flickr.com/photos/ul_marga/755378645/Monday, December 7, 2009



sharing knowledge

http://flickr.com/photos/kgoldenchild/2223459035/Monday, December 7, 2009



easier communicationhttp://flickr.com/photos/hchalkley/92120879/




easier integrationhttp://flickr.com/photos/stevehopson/1794895311




Data organizationwhen a large amount is available

http://flickr.com/photos/shetha/1561376494/Monday, December 7, 2009



Ontologiesin dialogue systems


Ontology in a dialog system

acoustic models

grammar

back-endsystem

DialogueManagement

Speech Recognition

Natural Language

Understanding

Natural Language

Generation

Speech Synthesis

User

lexicon

grammar

lexicon

grammar

prosody

coded speech

Domain Ontology


!"#"$%&''

()#*+)*"''

,-."/'

01*$)2'2-."/3''

4)3".'-#'

&-$5+3'.)6)'

("7%&-#'

8$-5"$'

0-+#39'

:6&;'

<#6-/-*='

<#6-/-*='

,)55%#*'

>+/"3'

?8@'A&63'

(B3'(B3'

C-$.3'C-$.3'

@5""&D''

>"&-*#%E"$'

8)$3"$'

@5""&D''

@=#6D"3%3'

@+$F)&"''

!"#"$)G-#'

H-2)%#1@5"&%I&'

?-25-#"#63'

J#6"$5$"6)G-#'

,)#)*"$!H%3&-+$3"''

@6)6"'

!$)22)$'

TRIPS Univ Rochester


McGuinness’ spectrum


My work so far...


OntoChef a cooking ontology

joint work with R. Ribeiro, F. Batista, N. Mamede and P. VazMonday, December 7, 2009

Recipe: Chocolate Cake

Ingredients

Procedure

4 oz of sweet dark chocolate

5 eggs

...

separate egg whites and egg yolks

with an electric mixer whisk the egg

whites until forming stiff peaks

put egg whites in a container

...

video file

...

Main goal


Competency QuestionsRecipes:

- How do I make recipe R? - What are the quantities to make recipe R for 4 persons?

Actions: - How do I do A1?

Times: - Which recipes that take less than 10 minutes to do?

Food: - Which recipes have food item F1, but not F2? - Which recipes have as main ingredient food item F1?

Utensils: - Which utensils are used in recipe R1? - Which recipes can be made using the microwave?

Equivalencies: - How many liters is a cup?


OntoChefcooking domain model


Food

Actions

Utensils

Equivalences

Recipes Units &

Measures

several modules


Formalization overview


Resulting Ontology4 main (reusable) modules:

- Food- Utensils- Actions- Recipes

3 auxiliary (reusable) modules: - Measures- Equivalences- Types of dishes

Metrics: - 1151 classes- 92 slots

- 52 relations between classes- 311 instances


OntoChefbuilding methodology


Main activities: - Knowledge acquisition- Conceptualization- Formalization- Evaluation


separate teams for bulk work

Food

Actions

Utensils

Equivalences

Recipes Units &

Measures


weekly brainstorm sessionsMonday, December 7, 2009

hardest: reach agreementMonday, December 7, 2009

OntoChefknowledge sources



professional websites

Web Site #Recipes #Photos

www.vaqueiro.pt 3136 2790

www.continente.pt 467 467

sabores.sapo.pt 2760 1038

comezainas.clix.pt 1646 0

www.entrepratos.com 189 51

Total: 8198 4346Recipes collected from professional websites.


Web Site #Recipes

paladares.do.sapo.pt 329

receitasdeculinaria.blogs.sapo.pt 10

culinaria.no.sapo.pt 236

Total: 575Recipes collected from personal blogs.

personal blogs


CMU’s Recipe Database<recipe version="1.1"><line><originaltext>1 (18 ounce) package small chocolate chip cookies</originaltext><annotation>create_ing(ing0, "1 (18 ounce) package small chocolate chip cookies")</annotation>

</line><line><originaltext>1 cup hot fudge topping</originaltext><annotation>create_ing(ing2, "1 cup hot fudge topping")</annotation>

</line>...<line><originaltext>Spread 3/4 cup fudge topping over crust. </originaltext><annotation>separate(ing2, ing12, "3/4 cup fudge", ing13, "1/4 cup fudge", "")</annotation>

</line>...


USENET Cookbook.RH MOD.RECIPES-SOURCE CHOC-CAKE-1 D "20 May 86" 1986.RZ "CHOCOLATE FANTASY CAKE" "A no-bake chocolate cake"This is a very simple, quick, no-bake cake..IH "Serves 6\-8".SH! ! CAKE.IG "2\(12 cups" "cream" "600 ml".IG "3 cups" "milk" "750 ml".IG "3 pkgs" "instant chocolate pudding".PH.SK 1Arrange a layer of biscuits at the bottom of a large baking pan.....SH RATING.I Difficulty:easy..I Time:10 minutes preparation, several hours chilling..I Precision:measure the ingredients..WRMike Trachtman, Weizmann Institute, Rehovot, Israel, [email protected]


OntoChefconceptualization


OntoChefdatabase population

joint work with T. Machado and N. MamedeMonday, December 7, 2009

Manual input


Recipes

(in text)

Pro

ces

s

Internet

e-Books

Cookbooks

Database(ontology intances)

OntoChef(cooking ontology)

Ontology-basedsemi-automatic inputthrough natural language processing


OntoChefevaluation


OntoCheffuture work


additional languagesMonday, December 7, 2009

Knowledge sourcesBook’s Cover Book’s Bibliographic Reference

English:“Ingredients”, by Loukie Werle, Ullmann Publishing, 2009, 384 pp.

Portuguese:“Ingredientes”, by Loukie Werle, Könemann, 2009, 384 pp.

English:“The Cooks' Bible”, by Le Cordon Bleu, Carroll & Brown Publishers, 2005, 352 pp.

Portuguese:“O Livro das Técnicas Culinárias”, by E. Treuille & J. Wright, Livros e Livros, 2006, 352 p.

“Dicionário Visual Português/Inglês”, Civilização Editora, 2006, 360 pgs.“Dicionário Visual Português/Espanhol”, Civilização Editora, 2006, 360 pgs.“Dicionário Visual Português/Françês”, Civilização Editora, 2006, 360 pgs.“Dicionário Visual Português/Italiano”, Civilização Editora, 2006, 360 pgs.“Dicionário Visual Português/Alemão”, Civilização Editora, 2006, 360 pgs.


http://www.wook.pt/authors/detail/id/29351

http://www.wook.pt/authors/detail/id/29351

Collected informationLanguages # pairs PT-EN

Portuguese — English Ingredients 21125927Portuguese — English Actions 1658 5927

Portuguese — English 21575927

Portuguese — Spanish 2157Portuguese — French 2157Portuguese — Italian 2157Portuguese — German 2157

Total: 14555Monday, December 7, 2009

wine information


Cooking Coach an ontology-based

coaching dialogue systems


Recipe description



Available ‘tasks’

- next- previous- repeat- how? (down)- continue (up)


Cooking Corpus



Corpus annotation

Food

Actions

Utensils

Equivalences

Recipes Units &

Measures


Cooking Coach next steps...


Catalog

/ ID

Terms /

Glossary

General

Logic

Complexity

with automated Reasoningwithout automated Reasoning

ThesaurusTerm

Hierarchy

Formal

Taxonomy

Formal

Instance

Frames

(props)

Value

Restrictions

Description

Logics

Explore semantic spectrum


Reasoning

‘Caracóis’ (snails): broiled animal, a cake with snail shape


Tackle different users


Different coaching styles


Pri

or

exp

eri

en

ce

wit

h

the

Dia

log

Syste

m

Knowledge of the Domain

(Cooking prior experience)

none some a lot

no

ne

so

me

a lo

t

System’s evaluation


MIT’s Counter


DFKI’s Smart Kitchen


Coaching another tasks


Questions

http://flickr.com/photos/jazz_dalek/1845705965/Monday, December 7, 2009



Thank you!

http://flickr.com/photos/42304632@N00/232004616/Monday, December 7, 2009



Joana Paulo Pardal

[email protected]

ladydoctorbug.blogspot.comwww.l2f.inesc-id.pt ⁄!joana


mailto:[email protected]

mailto:[email protected]

http://blog.centopeia.com




Cooking an ontology-based coaching dialogue system

Joana Paulo PardalQuality and Usability ColloquiumDecember 7th, 2009Berlin, Germany


Cooking an ontology-based spoken dialogue system

Technology

technical

end systemmonday

ontology

based

spoken

2009 monday

ist

december 7