Semantic Representation and Formal Transformation kevin knight usc/isi MURI meeting, CMU, Nov 3-4, 2011
Feb 23, 2016
Semantic Representation and Formal Transformation
kevin knightusc/isi
MURI meeting, CMU, Nov 3-4, 2011
Machine Translation
Phrase-based MT
Syntax-based MT
Meaning-based MTsourcestring
meaningrepresentation
targetstring
sourcestring
targetstring
sourcestring
sourcetree
targettree
targetstring
sourcetree
targettree
05
101520253035 NIST 2009 c2e
Meaning-based MT
• Too big for this MURI:– What content goes into the meaning
representation?• linguistics, annotation
– How are meaning representations probabilistically generated, transformed, scored, ranked?
• automata theory, efficient algorithms– How can a full MT system be built and tested?
• engineering, language modeling, features, training
Meaning-based MT
• Too big for this MURI:– What content goes into the meaning
representation?• linguistics, annotation
– How are meaning representations probabilistically generated, transformed, scored, ranked?
• automata theory, efficient algorithms– How can a full MT system be built and tested?
• engineering, language modeling, features, training
Meaning-based MT
• Too big for this MURI:– What content goes into the meaning
representation?• linguistics, annotation
– How are meaning representations probabilistically generated, transformed, scored, ranked?
• automata theory, efficient algorithms– How can a full MT system be built and tested?
• engineering, language modeling, features, trainingLanguage-independent theory.But driven by practical desires.
Automata Frameworks
• How to represent and manipulate linguistic representations?
• Linguistics, NLP, and Automata Theory used to be together (1960s, 70s)– Context-free grammars were invented to model human
language– Tree transducers were invented to model transformational
grammar• They drifted apart• Renewed connections around MT (this century)• Role: greatly simplify systems!
Finite-State Transducer (FST)
k
n
i
g
h
t
q k q2 *e*
q2 n q N
q i q AYq g q3 *e*
q4 t qfinal Tq3 h q4 *e*
Original input: Transformation:q k
n
i
g
h
t
FST
qq2
qfinalq3 q4
k : *e*
n : N
h : *e*
g : *e*t : T
i : AY
Finite-State (String) Transducer
q2 n
i
g
h
t
q k q2 *e*
q2 n q N
q i q AYq g q3 *e*
q4 t qfinal Tq3 h q4 *e*
Original input: Transformation:k
n
i
g
h
t
FST
qq2
qfinalq3 q4
k : *e*
n : N
h : *e*
g : *e*t : T
i : AY
Finite-State (String) Transducer
N
q i
g
h
t
q k q2 *e*
q2 n q N
q i q AYq g q3 *e*
q4 t qfinal Tq3 h q4 *e*
Original input: Transformation:k
n
i
g
h
t
FST
qq2
qfinalq3 q4
k : *e*
n : N
h : *e*
g : *e*t : T
i : AY
Finite-State (String) Transducer
q g
h
t
q k q2 *e*
q2 n q N
q i q AYq g q3 *e*
q4 t qfinal Tq3 h q4 *e*
AY
N
Original input: Transformation:k
n
i
g
h
t
FST
qq2
qfinalq3 q4
k : *e*
n : N
h : *e*
g : *e*t : T
i : AY
Finite-State (String) Transducer
q3 h
t
q k q2 *e*
q2 n q N
q i q AYq g q3 *e*
q4 t qfinal Tq3 h q4 *e*
AY
N
Original input: Transformation:k
n
i
g
h
t
FST
qq2
qfinalq3 q4
k : *e*
n : N
h : *e*
g : *e*t : T
i : AY
Finite-State (String) Transducer
q4 t
q k q2 *e*
q2 n q N
q i q AYq g q3 *e*
q4 t qfinal Tq3 h q4 *e*
AY
N
Original input: Transformation:k
n
i
g
h
t
FST
qq2
qfinalq3 q4
k : *e*
n : N
h : *e*
g : *e*t : T
i : AY
Finite-State (String) Transducer
q k q2 *e*
q2 n q N
q i q AYq g q3 *e*
q4 t qfinal Tq3 h q4 *e*
T
qfinal
AY
N
k
n
i
g
h
t
Original input: Transformation:
FST
qq2
qfinalq3 q4
k : *e*
n : N
h : *e*
g : *e*t : T
i : AY
Transliteration
Angela Knight
a n ji ra na i to
transliteration
Frequently occurring translation problem for languageswith different sound systems and character sets.(Japanese, Chinese, Arabic, Russian, English…)
Can’t be solved by dictionary lookup.
Transliteration
Angela KnightWFST
7 input symbols 13 output symbols
Transliteration
Angela Knight
WFST B
WFSA A
WFST D
AE N J EH L UH N AY T
WFST C a n j i r a n a i t o
WFST B
WFSA A
WFST D
WFST C a n j i r a n a i t o
AE N J IH R UH N AY TAH N J IH L UH N AY T OH
+ millions more
+ millions more
+ millions more
DECODE
General-Purpose Algorithms for String Automata
N-best … … paths through an WFSA (Viterbi, 1967; Eppstein, 1998)
EM training Forward-backward EM (Baum & Welch, 1971; Eisner 2001)
Determinization … … of weighted string acceptors (Mohri, 1997)
Intersection WFSA intersection
Application string WFST WFSA
Transducer composition WFST composition (Pereira & Riley, 1996)
General-purpose toolkit Carmel (Graehl & Knight 97), OpenFST (Google, via AT&T), ...
S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
Original input: Transformation:
q S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
Top-Down Tree Transducer(W. Rounds 1970; J. Thatcher 1970)
S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
Original input: Transformation:
q S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
Top-Down Tree Transducer(W. Rounds 1970; J. Thatcher 1970)
q S
x0:NP VP
s x0, wa, r x2, ga, q x1
x1:VBZ x2:NP
0.2
S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
Original input: Transformation:
s NP
PRO
he
q VBZ
enjoys
r NP
VBG
listening
VP
P
to
NP
SBAR
music
, ,
Top-Down Tree Transducer(W. Rounds 1970; J. Thatcher 1970)
, wa ,ga
S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
Original input: Transformation:
s NP
PRO
he
q VBZ
enjoys
r NP
VBG
listening
VP
P
to
NP
SBAR
music
, ,
Top-Down Tree Transducer(W. Rounds 1970; J. Thatcher 1970)
, wa ,ga
s NP
PRO
kare
he
0.7
S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
Original input: Transformation:
q VBZ
enjoys
r NP
VBG
listening
VP
P
to
NP
SBAR
music
,kare wa,
Top-Down Tree Transducer(W. Rounds 1970; J. Thatcher 1970)
, ,ga
S
NP VP
PRO
he
VBZ
enjoys
NP
VBG
listening
VP
P
to
NP
SBAR
music
kare kikuongaku owa daisuki desugano
Original input: Final output:
, , , , , , ,,
Top-Down Tree Transducer(W. Rounds 1970; J. Thatcher 1970)
Top-Down Tree Transducer
• Introduced by Rounds (1970) & Thatcher (1970)“Recent developments in the theory of automata have pointed
to an extension of the domain of definition of automata from strings to trees … parts of mathematical linguistics can be formalized easily in a tree-automaton setting …”
(Rounds 1970, “Mappings on Grammars and Trees”, Math. Systems Theory 4(3))
• Large theory literature– e.g., Gécseg & Steinby (1984), Comon et al (1997)
• Once again re-connecting with NLP practice– e.g., Knight & Graehl (2005), Galley et al (2004, 2006),
May & Knight (2006, 2010), Maletti et al (2009)
Tree Transducers Can be Extracted from Bilingual Data (Galley et al, 04)
i felt obliged to do my part
我 有 责任 尽 一份 力
TREE TRANSDUCER RULES:
VBD(felt) 有VBN(obliged) 责任VB(do) 尽NN(part) 一份NN(part) 一份 力VP-C(x0:VBN x1:SG-C) x0 x1VP(TO(to) x0:VP-C) x0 …S(x0:NP-C x1:VP) x0 x1
SNP-C VP VP-C VBD SG-C VP VBN TO VP-C VB NP-CNPB NPB
PRP PRP$ NN
这 7 人 中包括 来自 法国 和 俄罗斯 的 宇航 员 .
Syntax-Based Decoding
这 7 人 中包括 来自 法国 和 俄罗斯 的 宇航 员 .
RULE 1:DT(these) 这
RULE 2:VBP(include) 中包括
RULE 6:NNP(Russia) 俄罗斯
RULE 4:NNP(France) 法国
RULE 8:NP(NNS(astronauts)) 宇航 , 员
RULE 5:CC(and) 和
RULE 9:PUNC(.) .
“these” “Russia” “astronauts” “.”“include” “France” “and”
Syntax-Based Decoding
RULE 1:DT(these) 这
RULE 2:VBP(include) 中包括
RULE 6:NNP(Russia) 俄罗斯
RULE 4:NNP(France) 法国
RULE 8:NP(NNS(astronauts)) 宇航 , 员
RULE 5:CC(and) 和
RULE 9:PUNC(.) .
这 7 人 中包括 来自 法国 和 俄罗斯 的 宇航 员 .
RULE 13:NP(x0:NNP, x1:CC, x2:NNP) x0 , x1 , x2
“France and Russia”
“include”“these” “France” “and” “Russia” “astronauts” “.”
Syntax-Based Decoding
RULE 1:DT(these) 这
RULE 2:VBP(include) 中包括
RULE 6:NNP(Russia) 俄罗斯
RULE 4:NNP(France) 法国
RULE 8:NP(NNS(astronauts)) 宇航 , 员
RULE 5:CC(and) 和
RULE 9:PUNC(.) .
这 7 人 中包括 来自 法国 和 俄罗斯 的 宇航 员 .
RULE 13:NP(x0:NNP, x1:CC, x2:NNP) x0 , x1 , x2
RULE 11:VP(VBG(coming), PP(IN(from), x0:NP)) 来自 , x0
“France and Russia”
“coming from France and Russia”
“these” “Russia” “astronauts” “.”“include” “France” “&”
Syntax-Based Decoding
RULE 1:DT(these) 这
RULE 2:VBP(include) 中包括
RULE 6:NNP(Russia) 俄罗斯
RULE 4:NNP(France) 法国
RULE 8:NP(NNS(astronauts)) 宇航 , 员
RULE 5:CC(and) 和
RULE 9:PUNC(.) .
这 7 人 中包括 来自 法国 和 俄罗斯 的 宇航 员 .
RULE 13:NP(x0:NNP, x1:CC, x2:NNP) x0 , x1 , x2
RULE 11:VP(VBG(coming), PP(IN(from), x0:NP)) 来自 , x0
RULE 16:NP(x0:NP, x1:VP) x1 , 的 , x0
“astronauts coming fromFrance and Russia”
“France and Russia”
“coming from France and Russia”
“these” “Russia” “astronauts” “.”“include” “France” “&”
Syntax-Based Decoding
RULE 1:DT(these) 这
RULE 2:VBP(include) 中包括
RULE 6:NNP(Russia) 俄罗斯
RULE 4:NNP(France) 法国
RULE 8:NP(NNS(astronauts)) 宇航 , 员
RULE 5:CC(and) 和
RULE 9:PUNC(.) .
这 7 人 中包括 来自 法国 和 俄罗斯 的 宇航 员 .
RULE 13:NP(x0:NNP, x1:CC, x2:NNP) x0 , x1 , x2
RULE 16:NP(x0:NP, x1:VP) x1 , 的 , x0
RULE 11:VP(VBG(coming), PP(IN(from), x0:NP)) 来自 , x0
RULE 14:VP(x0:VBP, x1:NP) x0 , x1
“include astronauts coming fromFrance and Russia”
“France and Russia”
“coming from France and Russia”
“astronauts coming fromFrance and Russia”
“these” “Russia” “astronauts” “.”“include” “France” “&”
RULE 1:DT(these) 这
RULE 2:VBP(include) 中包括
RULE 6:NNP(Russia) 俄罗斯
RULE 4:NNP(France) 法国
RULE 8:NP(NNS(astronauts)) 宇航 , 员
RULE 5:CC(and) 和
RULE 9:PUNC(.) .
这 7 人 中包括 来自 法国 和 俄罗斯 的 宇航 员 .
RULE 10:NP(x0:DT, CD(7), NNS(people) x0 , 7 人
RULE 13:NP(x0:NNP, x1:CC, x2:NNP) x0 , x1 , x2
RULE 15:S(x0:NP, x1:VP, x2:PUNC) x0 , x1 , x2
RULE 16:NP(x0:NP, x1:VP) x1 , 的 , x0
RULE 11:VP(VBG(coming), PP(IN(from), x0:NP)) 来自 , x0
RULE 14:VP(x0:VBP, x1:NP) x0 , x1
“These 7 people include astronauts coming from France and Russia”
“France and Russia”
“coming from France and Russia”
“astronauts coming fromFrance and Russia”
“these 7 people”
“include astronauts coming fromFrance and Russia”
“these” “Russia” “astronauts” “.”“include” “France” “&”
These 7 people include astronauts coming from France and Russia .
DT CD VBP NNS IN NNP CC NNP PUNC
NPNP NP
VP
NP
VP
S
NNS VBG
PP
NPNP
DerivedEnglish Tree
General-Purpose Algorithms for Tree AutomataString Automata
AlgorithmsTree Automata
AlgorithmsN-best … … paths through an WFSA
(Viterbi, 1967; Eppstein, 1998)… trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)
EM training Forward-backward EM (Baum/Welch, 1971; Eisner 2003)
Tree transducer EM training (Graehl & Knight, 2004)
Determinization … … of weighted string acceptors (Mohri, 1997)
… of weighted tree acceptors (Borchardt & Vogler, 2003; May & Knight, 2005)
Intersection WFSA intersection Tree acceptor intersection
Applying transducers string WFST WFSA tree TT weighted tree acceptor
Transducer composition WFST composition (Pereira & Riley, 1996)
Many tree transducers not closed under composition (Maletti et al 09)
General-purpose tools Carmel, OpenFST Tiburon (May & Knight 10)
Machine Translation
Phrase-based MT
Syntax-based MT
Meaning-based MTsourcestring
meaningrepresentation
targetstring
sourcestring
targetstring
sourcestring
sourcetree
targettree
targetstring
sourcetree
targettree
Five Equivalent Meaning Representation Formats
(w / WANT :agent (b / BOY) :patient (g / GO :agent b)))
w, b, g : instance(w, WANT) ^ instance(g, GO) ^ instance(b, BOY) ^ agent(w, b) ^ patient(w, g) ^ agent(g, b)
E
WANT
BOY
GO
instance
instance
instanceagent
patient
agent
((x0 instance) = WANT((x1 instance) = BOY((x2 instance) = GO((x0 agent) = x1((x0 patent) = x2((x2 agent) = x1
instance: WANTagent:patient: instance: GO
agent:
instance: BOY1
1
LOGICAL FORM PATH EQUATIONS
FEATURE STRUCTURE
DIRECTED ACYCLIC GRAPH
PENMAN
“The boy wantsto go.”
Example“Government forces closed on rebel outposts on Thursday, showering the western mountain city of Zintan with missiles and attacking insurgents holed up near the Tunisian border, according to rebel sources.”
(s / say :agent (s2 / source :mod (r / rebel)) :patient (a / and :op1 (c / close-on :agent (f / force :mod (g / government)) :patient (o / outpost :mod (r2 / rebel)) :temporal-locating (t / thursday)) :op2 (s / shower :agent f :patient (c / city :mod (m / mountain) :mod (w / west) :name "Zintan") :instrument (m2 / missile)) :op3 (a / attack :agent f :patient (i / insurgent :agent-of (h / hole-up :pp-near (b / border :gpi (c2 / country :name "Tunisia"]
Slogan: “more logical than a parse tree”
General-Purpose Algorithms for Feature Structures (Graphs)String Automata
AlgorithmsTree Automata
AlgorithmsGraph Automata
Algorithms?N-best … … paths through an WFSA
(Viterbi, 1967; Eppstein, 1998)… trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)
EM training Forward-backward EM (Baum/Welch, 1971; Eisner 2003)
Tree transducer EM training (Graehl & Knight, 2004)
Determinization… … of weighted string acceptors (Mohri, 1997)
… of weighted tree acceptors (Borchardt & Vogler, 2003; May & Knight, 2005)
Intersection WFSA intersection Tree acceptor intersection
Applying transducers
string WFST WFSA tree TT weighted tree acceptor
Transducer composition
WFST composition (Pereira & Riley, 1996)
Many tree transducers not closed under composition (Maletti et al 09)
General tools Carmel, OpenFST Tiburon (May & Knight 10)
Automata Frameworks• Hyperedge-replacement graph grammars
– (Drewes et al)• DAG acceptors
– (Hart 75)• DAG-to-tree transducers
– (Kamimura & Slutski 82)
Mapping Between Meaning and Text
the boy wants to seeWANT
BOY
SEE
instance
instance
instanceagent
patient
agent
foreigntext
Mapping Between Meaning and Text
the boy wants to be seenWANT
BOY
SEE
instance
instance
instanceagent
patient
patient
foreigntext
Mapping Between Meaning and Text
the boy wants the girlto be seen
WANT
BOY
SEE
instance
instance
instanceagent
patient
patient
GIRL
instance
foreigntext
Mapping Between Meaning and Text
the boy wants to see the girlWANT
BOY
SEE
instance
instance
instanceagent
patient
patient
GIRL
instance
agent
foreigntext
Mapping Between Meaning and Text
the boy wants to see himself
WANT
BOY
SEE
instance
instance
instanceagent
patient
patientagent
foreigntext
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
PROMISE
BOY
GO
instance
instance
instanceagent
patient
recipient
GIRL
instance
agent
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
PROMISE
BOY
GO
instance
instance
instance
agent
patient
recipient
GIRL
instance
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
PROMISE
BOY
GO
instance
instance
instance
agent
patient
recipient
instance
q.NN | girl
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
PROMISE
GO
instance
instance
instance
agent
patient
recipient
instance
q.NN | girl
q.NN | boy
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
PROMISE
instance
instance
instance
agent
patient
recipient
instanceq.VBZ | goesq.NN
| girl
q.NN | boy
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
PROMISE
instance
instance
instance
agent
patient
recipient
instanceq.VB | goq.NN
| girl
q.NN | boy
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
instance
instance
instance
agent
patient
recipient
q.NN | girl
instanceq.VB | go
q.VBD | promised
q.NN | boy
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
instance
instance
agent
patient
recipient
q.VB | go
q.VBD | promised
q.NP | NN | boy
DT | the
q. NP | NN | girl
DT | the
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
instance
instance
patient
recipient
q.VB | go
q.VBD | promised
q.agt NP | NN | boy
DT | the
q. NP | NN | girl
DT | the
q.subj NP | NN | boy
DT | the
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
instance
patient
recipientq.VP / \TO VB | |to go
q.VBD | promised
q. NP | NN | girl
DT | the
q.agt NP | NN | boy
DT | the
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
instance
recipientqpatsubj.VP / \ TO VB | | to go
q.VBD | promised
q.agt NP | NN | boy
DT | the
q. NP | NN | girl
DT | the
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
instance
q.VBD | promised
qpatsubj.VP / \ TO VB | | to go
q.agt NP | NN | boy
DT | the
q.rec NP | NN | girl
DT | the
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
VBD | promised
VP / \ TO VB | | to go
S
NP | NN | boy
DT | the
NP | NN | girl
DT | the
VP
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
VBD | promised
VP / \ TO VB | | to go
S
NP | NN | boy
DT | the
NP | NN | girl
DT | the
VP
PROMISE
BOY
GO
instance
instance
agent
patient
recipient
GIRL
instance
agent
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
VBD | persuaded
S
NP | NN | boy
DT | the
NP | NN | girl
DT | the
VP
PERSUADE
BOY
GO
instance
instance
agent
patient
recipient
GIRL
instance
agent
VP / \ TO VB | | to go
Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)
VBD | persuaded
VP
that he would go
S
NP | NN | boy
DT | the
NP | NN | girl
DT | the
VP
PERSUADE
BOY
GO
instance
instance
agent
patient
recipient
GIRL
instance
agent
General-Purpose Algorithms for Feature Structures
String Automata Algorithms
Tree Automata Algorithms
Graph Automata Algorithms?
N-best … … paths through an WFSA (Viterbi, 1967; Eppstein, 1998)
… trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)
EM training Forward-backward EM (Baum/Welch, 1971; Eisner 2003)
Tree transducer EM training (Graehl & Knight, 2004)
Determinization… … of weighted string acceptors (Mohri, 1997)
… of weighted tree acceptors (Borchardt & Vogler, 2003; May & Knight, 2005)
Intersection WFSA intersection Tree acceptor intersection
Applying transducers
string WFST WFSA tree TT weighted tree acceptor
Transducer composition
WFST composition (Pereira & Riley, 1996)
Many tree transducers not closed under composition (Maletti et al 09)
General tools Carmel, OpenFST Tiburon (May & Knight 10)
Automata for Statistical MTUsed in SMT
Automata Framework Devised1960 1970 1980 1990 2000
1990
2000
2010
2020
2010
FST
TDTT
DAG
end